【AI达人特训营】基于ResNet50的NAM注意力机制论文复现

【AI达人特训营】NAM: Normalization-based Attention Module论文复现本项目源于百度AI达人训练营。通过论文的领读分析和代码解读，论文精读和飞桨（PaddlePaddle）代码复现相结合方式学习。一、论文解读摘要本文提出一种基于归一化的注意力模块（NAM），可以降低不太显著的特征的权重，这种方式在注意力模块上应用了稀疏的权重惩罚，这使得这些权重在计算上更加高效

AI Studio

809人浏览 · 2022-06-23 21:51:57

AI Studio · 2022-06-23 21:51:57 发布

【AI达人特训营】NAM: Normalization-based Attention Module论文复现
本项目源于百度AI达人训练营。通过论文的领读分析和代码解读，论文精读和飞桨（PaddlePaddle）代码复现相结合方式学习。

一、论文解读
摘要

本文提出一种基于归一化的注意力模块（NAM），可以降低不太显著的特征的权重，这种方式在注意力模块上应用了稀疏的权重惩罚，这使得这些权重在计算上更加高效，同时能够保持同样的性能。我们在ResNet和MobileNet上和其他的注意力方式进行了对比，我们的方法可以达到更高的准确率。

1.介绍

注意力机制在近年来大热，注意力机制可以帮助神经网络抑制通道中或者是空间中不太显著的特征。之前的很多的研究聚焦于如何通过注意力算子来获取显著性的特征。这些方法成功的发现了特征的不同维度之间的互信息量。但是，缺乏对权值的贡献因子的考虑，而这个贡献因子可以进一步的抑制不显著的特征。因此，我们瞄准了利用权值的贡献因子来提升注意力的效果。我们使用了Batch Normalization的缩放因子来表示权值的重要程度。这样可以避免如SE，BAM和CBAM一样增加全连接层和卷积层。这样，我们提出了一个新的注意力方式：基于归一化的注意力（NAM）

2.方法

我们提出的NAM是一种轻量级的高效的注意力机制，我们采用了CBAM的模块集成方式，重新设计了通道注意力和空间注意力子模块，这样，NAM可以嵌入到每个网络block的最后。对于残差网络，可以嵌入到残差结构的最后。对于通道注意力子模块，我们使用了Batch Normalization中的缩放因子，如式子（1），缩放因子反映出各个通道的变化的大小，也表示了该通道的重要性。为什么这么说呢，可以这样理解，缩放因子即BN中的方差，方差越大表示该通道变化的越厉害，那么该通道中包含的信息会越丰富，重要性也越大，而那些变化不大的通道，信息单一，重要性小。

因此，通道注意力子模块如图1，式（2），用表示最后得到的输出特征，γ是每个通道的缩放因子，因此，每个通道的权值可以得到，如果对空间中的每个像素使用同样的归一化方法，就可以得到空间注意力的权重，式（3），就叫做像素归一化。像素注意力见图2，在这里插入图片描述
为了抑制不重要的特征，我们在损失函数中加入了一个正则化项，如（4）式：
3.实验

我们将NAM和SE，BAM，CBAM，TAM在ResNet和MobileNet上，在CIFAR100数据集和ImageNet数据集上进行了对比，我们对每种注意力机制都使用了同样的预处理和训练方式，对比结果表示，在CIFAR100上，单独使用NAM的通道注意力或者空间注意力就可以达到超越其他方式的效果。在ImageNet上，同时使用NAM的通道注意力和空间注意力可以达到超越其他方法的效果。
在这里插入图片描述
4.结论

我们提出了一个NAM模块，该模块通过抑制不太显著的特征来提高效率。我们的实验表明，NAM在ResNet和MobileNet上都提供了效率增益。我们正在对NAM的集成变化和超参数调整性能进行详细分析。我们还计划使用不同的模型压缩技术优化NAM，以提高其效率。未来，我们将研究它对其他深度学习体系结构和应用程序的影响。

二、数据集介绍
CIFAR100数据集有100个类。每个类有600张大小为32 × 32 32\times 3232×32的彩色图像，其中500张作为训练集，100张作为测试集。对于每一张图像，它有fine_labels和coarse_labels两个标签，分别代表图像的细粒度和粗粒度标签，对应下图中的classes和superclass。也就是说，CIFAR100数据集是层次的。在这里插入图片描述
三、基于ResNet50的cirfar100实验
3.1 导入、划分数据集
In [8]
import paddle
import paddle.vision.transforms as t

def data_process():
# 数据增强策略
transform_strategy = t.Compose([
t.ColorJitter(), #随即调节亮度、对比度等
t.RandomHorizontalFlip(), # 随机水平翻转
t.RandomVerticalFlip(), # 随机垂直翻转
t.ToTensor() # 转化为张量
])

# 加载训练数据集
train_dataset = paddle.vision.datasets.Cifar100(
    mode='train',
    transform=transform_strategy
)

# 测试集采用与训练集相同的增强策略，检验模型的泛化能力
eval_dataset = paddle.vision.datasets.Cifar100(
    mode='test', 
    transform=transform_strategy
)

print('训练集样本数:', str(len(train_dataset)), '| 测试集样本数:', str(len(eval_dataset)))
return train_dataset, eval_dataset

train_dataset, eval_dataset = data_process() #获取数据
item 52/41261 […] - ETA: 1:25 - 2ms/item
Cache file /home/aistudio/.cache/paddle/dataset/cifar/cifar-100-python.tar.gz not found, downloading https://dataset.bj.bcebos.com/cifar/cifar-100-python.tar.gz
Begin to download
item 40921/41261 [============================>.] - ETA: 0s - 674us/item

Download finished
训练集样本数: 50000 | 测试集样本数: 10000
3.2 调用paddle API搭建resnet50
In [9]
model = paddle.Model(paddle.vision.models.resnet50(pretrained=False))
#模型可视化
model.summary((-1, 3, 32, 32))
W0623 11:42:45.144419 167 gpu_context.cc:278] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 10.1
W0623 11:42:45.149612 167 gpu_context.cc:306] device: 0, cuDNN Version: 7.6.

Layer (type) Input Shape Output Shape Param #

 Conv2D-1         [[1, 3, 32, 32]]     [1, 64, 16, 16]         9,408

BatchNorm2D-1 [[1, 64, 16, 16]] [1, 64, 16, 16] 256
ReLU-1 [[1, 64, 16, 16]] [1, 64, 16, 16] 0
MaxPool2D-1 [[1, 64, 16, 16]] [1, 64, 8, 8] 0
Conv2D-3 [[1, 64, 8, 8]] [1, 64, 8, 8] 4,096
BatchNorm2D-3 [[1, 64, 8, 8]] [1, 64, 8, 8] 256
ReLU-2 [[1, 256, 8, 8]] [1, 256, 8, 8] 0
Conv2D-4 [[1, 64, 8, 8]] [1, 64, 8, 8] 36,864
BatchNorm2D-4 [[1, 64, 8, 8]] [1, 64, 8, 8] 256
Conv2D-5 [[1, 64, 8, 8]] [1, 256, 8, 8] 16,384
BatchNorm2D-5 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
Conv2D-2 [[1, 64, 8, 8]] [1, 256, 8, 8] 16,384
BatchNorm2D-2 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
BottleneckBlock-1 [[1, 64, 8, 8]] [1, 256, 8, 8] 0
Conv2D-6 [[1, 256, 8, 8]] [1, 64, 8, 8] 16,384
BatchNorm2D-6 [[1, 64, 8, 8]] [1, 64, 8, 8] 256
ReLU-3 [[1, 256, 8, 8]] [1, 256, 8, 8] 0
Conv2D-7 [[1, 64, 8, 8]] [1, 64, 8, 8] 36,864
BatchNorm2D-7 [[1, 64, 8, 8]] [1, 64, 8, 8] 256
Conv2D-8 [[1, 64, 8, 8]] [1, 256, 8, 8] 16,384
BatchNorm2D-8 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
BottleneckBlock-2 [[1, 256, 8, 8]] [1, 256, 8, 8] 0
Conv2D-9 [[1, 256, 8, 8]] [1, 64, 8, 8] 16,384
BatchNorm2D-9 [[1, 64, 8, 8]] [1, 64, 8, 8] 256
ReLU-4 [[1, 256, 8, 8]] [1, 256, 8, 8] 0
Conv2D-10 [[1, 64, 8, 8]] [1, 64, 8, 8] 36,864
BatchNorm2D-10 [[1, 64, 8, 8]] [1, 64, 8, 8] 256
Conv2D-11 [[1, 64, 8, 8]] [1, 256, 8, 8] 16,384
BatchNorm2D-11 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
BottleneckBlock-3 [[1, 256, 8, 8]] [1, 256, 8, 8] 0
Conv2D-13 [[1, 256, 8, 8]] [1, 128, 8, 8] 32,768
BatchNorm2D-13 [[1, 128, 8, 8]] [1, 128, 8, 8] 512
ReLU-5 [[1, 512, 4, 4]] [1, 512, 4, 4] 0
Conv2D-14 [[1, 128, 8, 8]] [1, 128, 4, 4] 147,456
BatchNorm2D-14 [[1, 128, 4, 4]] [1, 128, 4, 4] 512
Conv2D-15 [[1, 128, 4, 4]] [1, 512, 4, 4] 65,536
BatchNorm2D-15 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,048
Conv2D-12 [[1, 256, 8, 8]] [1, 512, 4, 4] 131,072
BatchNorm2D-12 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,048
BottleneckBlock-4 [[1, 256, 8, 8]] [1, 512, 4, 4] 0
Conv2D-16 [[1, 512, 4, 4]] [1, 128, 4, 4] 65,536
BatchNorm2D-16 [[1, 128, 4, 4]] [1, 128, 4, 4] 512
ReLU-6 [[1, 512, 4, 4]] [1, 512, 4, 4] 0
Conv2D-17 [[1, 128, 4, 4]] [1, 128, 4, 4] 147,456
BatchNorm2D-17 [[1, 128, 4, 4]] [1, 128, 4, 4] 512
Conv2D-18 [[1, 128, 4, 4]] [1, 512, 4, 4] 65,536
BatchNorm2D-18 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,048
BottleneckBlock-5 [[1, 512, 4, 4]] [1, 512, 4, 4] 0
Conv2D-19 [[1, 512, 4, 4]] [1, 128, 4, 4] 65,536
BatchNorm2D-19 [[1, 128, 4, 4]] [1, 128, 4, 4] 512
ReLU-7 [[1, 512, 4, 4]] [1, 512, 4, 4] 0
Conv2D-20 [[1, 128, 4, 4]] [1, 128, 4, 4] 147,456
BatchNorm2D-20 [[1, 128, 4, 4]] [1, 128, 4, 4] 512
Conv2D-21 [[1, 128, 4, 4]] [1, 512, 4, 4] 65,536
BatchNorm2D-21 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,048
BottleneckBlock-6 [[1, 512, 4, 4]] [1, 512, 4, 4] 0
Conv2D-22 [[1, 512, 4, 4]] [1, 128, 4, 4] 65,536
BatchNorm2D-22 [[1, 128, 4, 4]] [1, 128, 4, 4] 512
ReLU-8 [[1, 512, 4, 4]] [1, 512, 4, 4] 0
Conv2D-23 [[1, 128, 4, 4]] [1, 128, 4, 4] 147,456
BatchNorm2D-23 [[1, 128, 4, 4]] [1, 128, 4, 4] 512
Conv2D-24 [[1, 128, 4, 4]] [1, 512, 4, 4] 65,536
BatchNorm2D-24 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,048
BottleneckBlock-7 [[1, 512, 4, 4]] [1, 512, 4, 4] 0
Conv2D-26 [[1, 512, 4, 4]] [1, 256, 4, 4] 131,072
BatchNorm2D-26 [[1, 256, 4, 4]] [1, 256, 4, 4] 1,024
ReLU-9 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-27 [[1, 256, 4, 4]] [1, 256, 2, 2] 589,824
BatchNorm2D-27 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
Conv2D-28 [[1, 256, 2, 2]] [1, 1024, 2, 2] 262,144
BatchNorm2D-28 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 4,096
Conv2D-25 [[1, 512, 4, 4]] [1, 1024, 2, 2] 524,288
BatchNorm2D-25 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 4,096
BottleneckBlock-8 [[1, 512, 4, 4]] [1, 1024, 2, 2] 0
Conv2D-29 [[1, 1024, 2, 2]] [1, 256, 2, 2] 262,144
BatchNorm2D-29 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
ReLU-10 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-30 [[1, 256, 2, 2]] [1, 256, 2, 2] 589,824
BatchNorm2D-30 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
Conv2D-31 [[1, 256, 2, 2]] [1, 1024, 2, 2] 262,144
BatchNorm2D-31 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 4,096
BottleneckBlock-9 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-32 [[1, 1024, 2, 2]] [1, 256, 2, 2] 262,144
BatchNorm2D-32 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
ReLU-11 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-33 [[1, 256, 2, 2]] [1, 256, 2, 2] 589,824
BatchNorm2D-33 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
Conv2D-34 [[1, 256, 2, 2]] [1, 1024, 2, 2] 262,144
BatchNorm2D-34 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 4,096
BottleneckBlock-10 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-35 [[1, 1024, 2, 2]] [1, 256, 2, 2] 262,144
BatchNorm2D-35 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
ReLU-12 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-36 [[1, 256, 2, 2]] [1, 256, 2, 2] 589,824
BatchNorm2D-36 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
Conv2D-37 [[1, 256, 2, 2]] [1, 1024, 2, 2] 262,144
BatchNorm2D-37 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 4,096
BottleneckBlock-11 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-38 [[1, 1024, 2, 2]] [1, 256, 2, 2] 262,144
BatchNorm2D-38 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
ReLU-13 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-39 [[1, 256, 2, 2]] [1, 256, 2, 2] 589,824
BatchNorm2D-39 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
Conv2D-40 [[1, 256, 2, 2]] [1, 1024, 2, 2] 262,144
BatchNorm2D-40 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 4,096
BottleneckBlock-12 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-41 [[1, 1024, 2, 2]] [1, 256, 2, 2] 262,144
BatchNorm2D-41 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
ReLU-14 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-42 [[1, 256, 2, 2]] [1, 256, 2, 2] 589,824
BatchNorm2D-42 [[1, 256, 2, 2]] [1, 256, 2, 2] 1,024
Conv2D-43 [[1, 256, 2, 2]] [1, 1024, 2, 2] 262,144
BatchNorm2D-43 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 4,096
BottleneckBlock-13 [[1, 1024, 2, 2]] [1, 1024, 2, 2] 0
Conv2D-45 [[1, 1024, 2, 2]] [1, 512, 2, 2] 524,288
BatchNorm2D-45 [[1, 512, 2, 2]] [1, 512, 2, 2] 2,048
ReLU-15 [[1, 2048, 1, 1]] [1, 2048, 1, 1] 0
Conv2D-46 [[1, 512, 2, 2]] [1, 512, 1, 1] 2,359,296
BatchNorm2D-46 [[1, 512, 1, 1]] [1, 512, 1, 1] 2,048
Conv2D-47 [[1, 512, 1, 1]] [1, 2048, 1, 1] 1,048,576
BatchNorm2D-47 [[1, 2048, 1, 1]] [1, 2048, 1, 1] 8,192
Conv2D-44 [[1, 1024, 2, 2]] [1, 2048, 1, 1] 2,097,152
BatchNorm2D-44 [[1, 2048, 1, 1]] [1, 2048, 1, 1] 8,192
BottleneckBlock-14 [[1, 1024, 2, 2]] [1, 2048, 1, 1] 0
Conv2D-48 [[1, 2048, 1, 1]] [1, 512, 1, 1] 1,048,576
BatchNorm2D-48 [[1, 512, 1, 1]] [1, 512, 1, 1] 2,048
ReLU-16 [[1, 2048, 1, 1]] [1, 2048, 1, 1] 0
Conv2D-49 [[1, 512, 1, 1]] [1, 512, 1, 1] 2,359,296
BatchNorm2D-49 [[1, 512, 1, 1]] [1, 512, 1, 1] 2,048
Conv2D-50 [[1, 512, 1, 1]] [1, 2048, 1, 1] 1,048,576
BatchNorm2D-50 [[1, 2048, 1, 1]] [1, 2048, 1, 1] 8,192
BottleneckBlock-15 [[1, 2048, 1, 1]] [1, 2048, 1, 1] 0
Conv2D-51 [[1, 2048, 1, 1]] [1, 512, 1, 1] 1,048,576
BatchNorm2D-51 [[1, 512, 1, 1]] [1, 512, 1, 1] 2,048
ReLU-17 [[1, 2048, 1, 1]] [1, 2048, 1, 1] 0
Conv2D-52 [[1, 512, 1, 1]] [1, 512, 1, 1] 2,359,296
BatchNorm2D-52 [[1, 512, 1, 1]] [1, 512, 1, 1] 2,048
Conv2D-53 [[1, 512, 1, 1]] [1, 2048, 1, 1] 1,048,576
BatchNorm2D-53 [[1, 2048, 1, 1]] [1, 2048, 1, 1] 8,192
BottleneckBlock-16 [[1, 2048, 1, 1]] [1, 2048, 1, 1] 0
AdaptiveAvgPool2D-1 [[1, 2048, 1, 1]] [1, 2048, 1, 1] 0
Linear-1 [[1, 2048]] [1, 1000] 2,049,000

Total params: 25,610,152
Trainable params: 25,503,912
Non-trainable params: 106,240

Input size (MB): 0.01
Forward/backward pass size (MB): 5.36
Params size (MB): 97.69
Estimated Total Size (MB): 103.07

{‘total_params’: 25610152, ‘trainable_params’: 25503912}
3.3 模型训练
In [12]
from paddle.optimizer.lr import CosineAnnealingDecay, MultiStepDecay, LinearWarmup

model.prepare(paddle.optimizer.SGD(learning_rate=0.001, parameters=model.parameters()),#使用Adam优化器，学习率为0.0001

paddle.nn.CrossEntropyLoss(),#损失函数使用交叉熵函数

paddle.metric.Accuracy())#Acc用top1与top5精准度表示

model.prepare(
paddle.optimizer.Momentum(
learning_rate=LinearWarmup(CosineAnnealingDecay(0.001, 100), 2000, 0., 0.001),
momentum=0.9,
parameters=model.parameters(),
weight_decay=5e-4),
paddle.nn.CrossEntropyLoss(),
paddle.metric.Accuracy(topk=(1,5)))

callback_visualdl = paddle.callbacks.VisualDL(log_dir=‘visualdl_log_dir’)
#开始模型训练
model.fit(train_dataset,
eval_dataset,
epochs=100,#训练的轮数
batch_size=128,#每次训练多少个
verbose=1,#显示模式
shuffle=True,#打乱数据集顺序
num_workers=4,
callbacks=callback_visualdl,
)
The loss value printed in the log is the current step, and the metric is the average value of previous steps.
Epoch 1/100
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/norm.py:654: UserWarning: When training, we now always track global mean and variance.
“When training, we now always track global mean and variance.”)
step 391/391 [] - loss: 5.8216 - acc_top1: 0.0083 - acc_top5: 0.0365 - 51ms/step
Eval begin…
step 79/79 [] - loss: 5.1818 - acc_top1: 0.0139 - acc_top5: 0.0630 - 30ms/step
Eval samples: 10000
Epoch 2/100
step 391/391 [] - loss: 4.9121 - acc_top1: 0.0193 - acc_top5: 0.0816 - 51ms/step
Eval begin…
step 79/79 [] - loss: 4.3763 - acc_top1: 0.0310 - acc_top5: 0.1156 - 29ms/step
Eval samples: 10000
Epoch 3/100
step 391/391 [] - loss: 4.5889 - acc_top1: 0.0426 - acc_top5: 0.1608 - 51ms/step
Eval begin…
step 79/79 [] - loss: 4.3340 - acc_top1: 0.0639 - acc_top5: 0.2092 - 29ms/step
Eval samples: 10000
Epoch 4/100
step 391/391 [] - loss: 4.0951 - acc_top1: 0.0792 - acc_top5: 0.2558 - 51ms/step
Eval begin…
step 79/79 [] - loss: 4.4047 - acc_top1: 0.0998 - acc_top5: 0.2908 - 29ms/step
Eval samples: 10000
Epoch 5/100
step 391/391 [] - loss: 4.0470 - acc_top1: 0.1093 - acc_top5: 0.3159 - 51ms/step
Eval begin…
step 79/79 [] - loss: 4.1264 - acc_top1: 0.1306 - acc_top5: 0.3494 - 29ms/step
Eval samples: 10000
Epoch 6/100
step 391/391 [] - loss: 3.3129 - acc_top1: 0.1412 - acc_top5: 0.3718 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.7290 - acc_top1: 0.1552 - acc_top5: 0.3856 - 30ms/step
Eval samples: 10000
Epoch 7/100
step 391/391 [] - loss: 3.6535 - acc_top1: 0.1597 - acc_top5: 0.4010 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.6085 - acc_top1: 0.1640 - acc_top5: 0.4079 - 29ms/step
Eval samples: 10000
Epoch 8/100
step 391/391 [] - loss: 3.2932 - acc_top1: 0.1744 - acc_top5: 0.4265 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.4222 - acc_top1: 0.1731 - acc_top5: 0.4195 - 31ms/step
Eval samples: 10000
Epoch 9/100
step 391/391 [] - loss: 3.4369 - acc_top1: 0.1843 - acc_top5: 0.4422 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.2243 - acc_top1: 0.1849 - acc_top5: 0.4386 - 29ms/step
Eval samples: 10000
Epoch 10/100
step 391/391 [] - loss: 3.5167 - acc_top1: 0.1973 - acc_top5: 0.4603 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.5050 - acc_top1: 0.1892 - acc_top5: 0.4476 - 29ms/step
Eval samples: 10000
Epoch 11/100
step 391/391 [] - loss: 3.1014 - acc_top1: 0.2053 - acc_top5: 0.4726 - 52ms/step
Eval begin…
step 79/79 [] - loss: 3.4008 - acc_top1: 0.1954 - acc_top5: 0.4587 - 29ms/step loss: 3.3302 - acc_top1: 0.1945 - acc_top5: 0.4608 - ETA: 0s - 32m
Eval samples: 10000
Epoch 12/100
step 391/391 [] - loss: 3.7199 - acc_top1: 0.2150 - acc_top5: 0.4872 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.4300 - acc_top1: 0.2004 - acc_top5: 0.4636 - 29ms/step loss: 3.0789 - acc_top1: 0.2027 - acc_top5: 0.4613 - ETA: 2s
Eval samples: 10000
Epoch 13/100
step 391/391 [] - loss: 3.4431 - acc_top1: 0.2238 - acc_top5: 0.4971 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.5822 - acc_top1: 0.2085 - acc_top5: 0.4721 - 38ms/step
Eval samples: 10000
Epoch 14/100
step 391/391 [] - loss: 2.7427 - acc_top1: 0.2316 - acc_top5: 0.5125 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.4022 - acc_top1: 0.2151 - acc_top5: 0.4802 - 29ms/step
Eval samples: 10000
Epoch 15/100
step 391/391 [] - loss: 3.3946 - acc_top1: 0.2384 - acc_top5: 0.5218 - 50ms/step
Eval begin…
step 79/79 [] - loss: 3.3966 - acc_top1: 0.2138 - acc_top5: 0.4899 - 29ms/step
Eval samples: 10000
Epoch 16/100
step 391/391 [] - loss: 3.0503 - acc_top1: 0.2481 - acc_top5: 0.5355 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.5413 - acc_top1: 0.2195 - acc_top5: 0.4901 - 32ms/step
Eval samples: 10000
Epoch 17/100
step 391/391 [] - loss: 3.1893 - acc_top1: 0.2571 - acc_top5: 0.5454 - 50ms/step
Eval begin…
step 79/79 [] - loss: 3.5470 - acc_top1: 0.2124 - acc_top5: 0.4812 - 29ms/step
Eval samples: 10000
Epoch 18/100
step 391/391 [] - loss: 2.9919 - acc_top1: 0.2657 - acc_top5: 0.5556 - 50ms/step
Eval begin…
step 79/79 [] - loss: 3.6241 - acc_top1: 0.2206 - acc_top5: 0.4934 - 29ms/step
Eval samples: 10000
Epoch 19/100
step 391/391 [] - loss: 2.9012 - acc_top1: 0.2698 - acc_top5: 0.5641 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.2880 - acc_top1: 0.2222 - acc_top5: 0.4976 - 29ms/step
Eval samples: 10000
Epoch 20/100
step 391/391 [] - loss: 2.8051 - acc_top1: 0.2770 - acc_top5: 0.5716 - 50ms/step
Eval begin…
step 79/79 [] - loss: 3.3445 - acc_top1: 0.2281 - acc_top5: 0.5019 - 29ms/step
Eval samples: 10000
Epoch 21/100
step 391/391 [] - loss: 2.8484 - acc_top1: 0.2824 - acc_top5: 0.5814 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.1611 - acc_top1: 0.2339 - acc_top5: 0.5085 - 29ms/step
Eval samples: 10000
Epoch 22/100
step 391/391 [] - loss: 2.6754 - acc_top1: 0.2907 - acc_top5: 0.5896 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.0240 - acc_top1: 0.2327 - acc_top5: 0.5114 - 30ms/step
Eval samples: 10000
Epoch 23/100
step 391/391 [] - loss: 3.2871 - acc_top1: 0.2964 - acc_top5: 0.5974 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.1273 - acc_top1: 0.2442 - acc_top5: 0.5140 - 30ms/step
Eval samples: 10000
Epoch 24/100
step 391/391 [] - loss: 3.0351 - acc_top1: 0.3056 - acc_top5: 0.6029 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.3547 - acc_top1: 0.2479 - acc_top5: 0.5299 - 29ms/step
Eval samples: 10000
Epoch 25/100
step 391/391 [] - loss: 2.7127 - acc_top1: 0.3093 - acc_top5: 0.6148 - 52ms/step
Eval begin…
step 79/79 [] - loss: 3.2127 - acc_top1: 0.2506 - acc_top5: 0.5291 - 30ms/step
Eval samples: 10000
Epoch 26/100
step 391/391 [] - loss: 2.8931 - acc_top1: 0.3145 - acc_top5: 0.6212 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9948 - acc_top1: 0.2512 - acc_top5: 0.5316 - 30ms/step
Eval samples: 10000
Epoch 27/100
step 391/391 [] - loss: 3.0113 - acc_top1: 0.3210 - acc_top5: 0.6290 - 52ms/step
Eval begin…
step 79/79 [] - loss: 3.1367 - acc_top1: 0.2537 - acc_top5: 0.5331 - 37ms/step
Eval samples: 10000
Epoch 28/100
step 391/391 [] - loss: 2.8563 - acc_top1: 0.3280 - acc_top5: 0.6368 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9553 - acc_top1: 0.2667 - acc_top5: 0.5432 - 29ms/step
Eval samples: 10000
Epoch 29/100
step 391/391 [] - loss: 2.3831 - acc_top1: 0.3327 - acc_top5: 0.6425 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.8827 - acc_top1: 0.2671 - acc_top5: 0.5509 - 30ms/step
Eval samples: 10000
Epoch 30/100
step 391/391 [] - loss: 2.8241 - acc_top1: 0.3421 - acc_top5: 0.6498 - 52ms/step
Eval begin…
step 79/79 [] - loss: 3.1073 - acc_top1: 0.2667 - acc_top5: 0.5525 - 30ms/step
Eval samples: 10000
Epoch 31/100
step 391/391 [] - loss: 2.5978 - acc_top1: 0.3466 - acc_top5: 0.6570 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.1769 - acc_top1: 0.2712 - acc_top5: 0.5555 - 30ms/step
Eval samples: 10000
Epoch 32/100
step 391/391 [] - loss: 2.6507 - acc_top1: 0.3562 - acc_top5: 0.6653 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.0157 - acc_top1: 0.2752 - acc_top5: 0.5587 - 30ms/step
Eval samples: 10000
Epoch 33/100
step 391/391 [] - loss: 3.0660 - acc_top1: 0.3567 - acc_top5: 0.6712 - 52ms/step
Eval begin…
step 79/79 [] - loss: 3.0034 - acc_top1: 0.2762 - acc_top5: 0.5559 - 30ms/step
Eval samples: 10000
Epoch 34/100
step 391/391 [] - loss: 2.4485 - acc_top1: 0.3656 - acc_top5: 0.6777 - 52ms/step
Eval begin…
step 79/79 [] - loss: 2.9226 - acc_top1: 0.2772 - acc_top5: 0.5563 - 29ms/step
Eval samples: 10000
Epoch 35/100
step 391/391 [] - loss: 2.7170 - acc_top1: 0.3770 - acc_top5: 0.6829 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.1584 - acc_top1: 0.2801 - acc_top5: 0.5603 - 29ms/step
Eval samples: 10000
Epoch 36/100
step 391/391 [] - loss: 2.3328 - acc_top1: 0.3813 - acc_top5: 0.6928 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.8032 - acc_top1: 0.2761 - acc_top5: 0.5617 - 30ms/step
Eval samples: 10000
Epoch 37/100
step 391/391 [] - loss: 2.3197 - acc_top1: 0.3866 - acc_top5: 0.6992 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9066 - acc_top1: 0.2725 - acc_top5: 0.5597 - 30ms/step
Eval samples: 10000
Epoch 38/100
step 391/391 [] - loss: 2.4766 - acc_top1: 0.3977 - acc_top5: 0.7053 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.7028 - acc_top1: 0.2792 - acc_top5: 0.5652 - 29ms/step
Eval samples: 10000
Epoch 39/100
step 391/391 [] - loss: 2.4647 - acc_top1: 0.4030 - acc_top5: 0.7120 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.8679 - acc_top1: 0.2783 - acc_top5: 0.5623 - 30ms/step
Eval samples: 10000
Epoch 40/100
step 391/391 [] - loss: 2.3726 - acc_top1: 0.4081 - acc_top5: 0.7183 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.6993 - acc_top1: 0.2803 - acc_top5: 0.5579 - 30ms/step
Eval samples: 10000
Epoch 41/100
step 391/391 [] - loss: 2.3839 - acc_top1: 0.4168 - acc_top5: 0.7266 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.0042 - acc_top1: 0.2769 - acc_top5: 0.5667 - 33ms/step
Eval samples: 10000
Epoch 42/100
step 391/391 [] - loss: 2.2273 - acc_top1: 0.4213 - acc_top5: 0.7319 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9708 - acc_top1: 0.2747 - acc_top5: 0.5610 - 29ms/step
Eval samples: 10000
Epoch 43/100
step 391/391 [] - loss: 2.3523 - acc_top1: 0.4268 - acc_top5: 0.7353 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.5551 - acc_top1: 0.2726 - acc_top5: 0.5633 - 29ms/step
Eval samples: 10000
Epoch 44/100
step 391/391 [] - loss: 2.4685 - acc_top1: 0.4367 - acc_top5: 0.7454 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.4999 - acc_top1: 0.2706 - acc_top5: 0.5608 - 29ms/step
Eval samples: 10000
Epoch 45/100
step 391/391 [] - loss: 2.3961 - acc_top1: 0.4377 - acc_top5: 0.7449 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.3164 - acc_top1: 0.2735 - acc_top5: 0.5627 - 32ms/step
Eval samples: 10000
Epoch 46/100
step 391/391 [] - loss: 2.4653 - acc_top1: 0.4486 - acc_top5: 0.7549 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9869 - acc_top1: 0.2840 - acc_top5: 0.5694 - 29ms/step
Eval samples: 10000
Epoch 47/100
step 391/391 [] - loss: 2.2756 - acc_top1: 0.4484 - acc_top5: 0.7588 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.1067 - acc_top1: 0.2838 - acc_top5: 0.5716 - 29ms/step
Eval samples: 10000
Epoch 48/100
step 391/391 [] - loss: 2.4643 - acc_top1: 0.4553 - acc_top5: 0.7622 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9304 - acc_top1: 0.2933 - acc_top5: 0.5712 - 30ms/step
Eval samples: 10000
Epoch 49/100
step 391/391 [] - loss: 2.4657 - acc_top1: 0.4611 - acc_top5: 0.7706 - 52ms/step
Eval begin…
step 79/79 [] - loss: 2.7275 - acc_top1: 0.2974 - acc_top5: 0.5844 - 30ms/step
Eval samples: 10000
Epoch 50/100
step 391/391 [] - loss: 2.1496 - acc_top1: 0.4677 - acc_top5: 0.7758 - 52ms/step
Eval begin…
step 79/79 [] - loss: 2.8040 - acc_top1: 0.2993 - acc_top5: 0.5862 - 29ms/step
Eval samples: 10000
Epoch 51/100
step 391/391 [] - loss: 2.0272 - acc_top1: 0.4741 - acc_top5: 0.7796 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.4976 - acc_top1: 0.2983 - acc_top5: 0.5870 - 29ms/step
Eval samples: 10000
Epoch 52/100
step 391/391 [] - loss: 1.9062 - acc_top1: 0.4817 - acc_top5: 0.7860 - 50ms/step
Eval begin…
step 79/79 [] - loss: 2.1902 - acc_top1: 0.3037 - acc_top5: 0.5881 - 29ms/step
Eval samples: 10000
Epoch 53/100
step 391/391 [] - loss: 2.2504 - acc_top1: 0.4873 - acc_top5: 0.7908 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.2911 - acc_top1: 0.3053 - acc_top5: 0.5894 - 29ms/step
Eval samples: 10000
Epoch 54/100
step 391/391 [] - loss: 2.3077 - acc_top1: 0.4942 - acc_top5: 0.7972 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.8821 - acc_top1: 0.3028 - acc_top5: 0.5855 - 29ms/step
Eval samples: 10000
Epoch 55/100
step 391/391 [] - loss: 1.6972 - acc_top1: 0.5009 - acc_top5: 0.7999 - 50ms/step
Eval begin…
step 79/79 [] - loss: 2.8060 - acc_top1: 0.2986 - acc_top5: 0.5821 - 37ms/step
Eval samples: 10000
Epoch 56/100
step 391/391 [] - loss: 1.7615 - acc_top1: 0.5104 - acc_top5: 0.8059 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9240 - acc_top1: 0.3039 - acc_top5: 0.5941 - 29ms/step
Eval samples: 10000
Epoch 57/100
step 391/391 [] - loss: 1.8560 - acc_top1: 0.5171 - acc_top5: 0.8115 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.8036 - acc_top1: 0.3018 - acc_top5: 0.5895 - 29ms/step
Eval samples: 10000
Epoch 58/100
step 391/391 [] - loss: 1.7631 - acc_top1: 0.5235 - acc_top5: 0.8165 - 50ms/step
Eval begin…
step 79/79 [] - loss: 2.8598 - acc_top1: 0.2962 - acc_top5: 0.5827 - 29ms/step
Eval samples: 10000
Epoch 59/100
step 391/391 [] - loss: 1.7417 - acc_top1: 0.5279 - acc_top5: 0.8220 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.2199 - acc_top1: 0.2978 - acc_top5: 0.5896 - 30ms/step
Eval samples: 10000
Epoch 60/100
step 391/391 [] - loss: 1.7674 - acc_top1: 0.5393 - acc_top5: 0.8285 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.8630 - acc_top1: 0.2959 - acc_top5: 0.5795 - 30ms/step
Eval samples: 10000
Epoch 61/100
step 391/391 [] - loss: 1.3035 - acc_top1: 0.5450 - acc_top5: 0.8302 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9358 - acc_top1: 0.2996 - acc_top5: 0.5866 - 29ms/step
Eval samples: 10000
Epoch 62/100
step 391/391 [] - loss: 1.7947 - acc_top1: 0.5515 - acc_top5: 0.8387 - 52ms/step
Eval begin…
step 79/79 [] - loss: 2.7642 - acc_top1: 0.2944 - acc_top5: 0.5785 - 30ms/step
Eval samples: 10000
Epoch 63/100
step 391/391 [] - loss: 1.7934 - acc_top1: 0.5578 - acc_top5: 0.8427 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.2590 - acc_top1: 0.2904 - acc_top5: 0.5701 - 32ms/step
Eval samples: 10000
Epoch 64/100
step 391/391 [] - loss: 1.9037 - acc_top1: 0.5635 - acc_top5: 0.8449 - 52ms/step
Eval begin…
step 79/79 [] - loss: 2.9783 - acc_top1: 0.2995 - acc_top5: 0.5813 - 30ms/step
Eval samples: 10000
Epoch 65/100
step 391/391 [] - loss: 1.8681 - acc_top1: 0.5721 - acc_top5: 0.8486 - 52ms/step
Eval begin…
step 79/79 [] - loss: 2.9527 - acc_top1: 0.2814 - acc_top5: 0.5709 - 30ms/step
Eval samples: 10000
Epoch 66/100
step 391/391 [] - loss: 1.8873 - acc_top1: 0.5746 - acc_top5: 0.8537 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.7710 - acc_top1: 0.2848 - acc_top5: 0.5751 - 30ms/step
Eval samples: 10000
Epoch 67/100
step 391/391 [] - loss: 1.6490 - acc_top1: 0.5793 - acc_top5: 0.8578 - 52ms/step
Eval begin…
step 79/79 [] - loss: 2.9304 - acc_top1: 0.2981 - acc_top5: 0.5861 - 30ms/step
Eval samples: 10000
Epoch 68/100
step 391/391 [] - loss: 1.5646 - acc_top1: 0.5872 - acc_top5: 0.8634 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.7347 - acc_top1: 0.2951 - acc_top5: 0.5789 - 30ms/step
Eval samples: 10000
Epoch 69/100
step 391/391 [] - loss: 1.3484 - acc_top1: 0.5881 - acc_top5: 0.8630 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.0181 - acc_top1: 0.3066 - acc_top5: 0.5908 - 32ms/step
Eval samples: 10000
Epoch 70/100
step 391/391 [] - loss: 1.4429 - acc_top1: 0.5969 - acc_top5: 0.8682 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9097 - acc_top1: 0.3112 - acc_top5: 0.5959 - 29ms/step
Eval samples: 10000
Epoch 71/100
step 391/391 [] - loss: 1.4620 - acc_top1: 0.6028 - acc_top5: 0.8715 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.6693 - acc_top1: 0.3119 - acc_top5: 0.5972 - 29ms/step
Eval samples: 10000
Epoch 72/100
step 391/391 [] - loss: 1.4809 - acc_top1: 0.6091 - acc_top5: 0.8749 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.6038 - acc_top1: 0.3128 - acc_top5: 0.5888 - 30ms/step
Eval samples: 10000
Epoch 73/100
step 391/391 [] - loss: 1.4798 - acc_top1: 0.6139 - acc_top5: 0.8800 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.7522 - acc_top1: 0.3162 - acc_top5: 0.5978 - 29ms/step
Eval samples: 10000
Epoch 74/100
step 391/391 [] - loss: 1.2607 - acc_top1: 0.6219 - acc_top5: 0.8848 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.3127 - acc_top1: 0.3183 - acc_top5: 0.5982 - 31ms/step
Eval samples: 10000
Epoch 75/100
step 391/391 [] - loss: 1.1673 - acc_top1: 0.6318 - acc_top5: 0.8888 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.8425 - acc_top1: 0.3115 - acc_top5: 0.6030 - 29ms/step
Eval samples: 10000
Epoch 76/100
step 391/391 [] - loss: 1.4403 - acc_top1: 0.6363 - acc_top5: 0.8925 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9370 - acc_top1: 0.3217 - acc_top5: 0.5940 - 30ms/step
Eval samples: 10000
Epoch 77/100
step 391/391 [] - loss: 1.3236 - acc_top1: 0.6443 - acc_top5: 0.8987 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.0327 - acc_top1: 0.3128 - acc_top5: 0.5929 - 29ms/step
Eval samples: 10000
Epoch 78/100
step 391/391 [] - loss: 1.7846 - acc_top1: 0.6460 - acc_top5: 0.9004 - 52ms/step
Eval begin…
step 79/79 [] - loss: 2.9746 - acc_top1: 0.3133 - acc_top5: 0.6020 - 30ms/step
Eval samples: 10000
Epoch 79/100
step 391/391 [] - loss: 1.1664 - acc_top1: 0.6522 - acc_top5: 0.9030 - 50ms/step
Eval begin…
step 79/79 [] - loss: 2.9526 - acc_top1: 0.3175 - acc_top5: 0.5957 - 29ms/step
Eval samples: 10000
Epoch 80/100
step 391/391 [] - loss: 1.3212 - acc_top1: 0.6623 - acc_top5: 0.9088 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.6803 - acc_top1: 0.3171 - acc_top5: 0.5908 - 30ms/step
Eval samples: 10000
Epoch 81/100
step 391/391 [] - loss: 0.9561 - acc_top1: 0.6657 - acc_top5: 0.9101 - 52ms/step
Eval begin…
step 79/79 [] - loss: 3.2766 - acc_top1: 0.3122 - acc_top5: 0.5888 - 29ms/step
Eval samples: 10000
Epoch 82/100
step 391/391 [] - loss: 1.2105 - acc_top1: 0.6770 - acc_top5: 0.9150 - 50ms/step
Eval begin…
step 79/79 [] - loss: 2.8955 - acc_top1: 0.3038 - acc_top5: 0.5915 - 29ms/step
Eval samples: 10000
Epoch 83/100
step 391/391 [] - loss: 1.2193 - acc_top1: 0.6803 - acc_top5: 0.9179 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.8806 - acc_top1: 0.3011 - acc_top5: 0.5742 - 32ms/step
Eval samples: 10000
Epoch 84/100
step 391/391 [] - loss: 1.3346 - acc_top1: 0.6901 - acc_top5: 0.9241 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.2308 - acc_top1: 0.3176 - acc_top5: 0.5958 - 29ms/step
Eval samples: 10000
Epoch 85/100
step 391/391 [] - loss: 1.2272 - acc_top1: 0.6933 - acc_top5: 0.9232 - 50ms/step
Eval begin…
step 79/79 [] - loss: 3.7281 - acc_top1: 0.2976 - acc_top5: 0.5796 - 29ms/step
Eval samples: 10000
Epoch 86/100
step 391/391 [] - loss: 1.4050 - acc_top1: 0.6962 - acc_top5: 0.9265 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.7497 - acc_top1: 0.3092 - acc_top5: 0.5891 - 30ms/step
Eval samples: 10000
Epoch 87/100
step 391/391 [] - loss: 1.1955 - acc_top1: 0.7055 - acc_top5: 0.9302 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9081 - acc_top1: 0.2900 - acc_top5: 0.5724 - 30ms/step
Eval samples: 10000
Epoch 88/100
step 391/391 [] - loss: 1.1573 - acc_top1: 0.7063 - acc_top5: 0.9303 - 50ms/step
Eval begin…
step 79/79 [] - loss: 3.2307 - acc_top1: 0.3017 - acc_top5: 0.5833 - 29ms/step
Eval samples: 10000
Epoch 89/100
step 391/391 [] - loss: 1.0924 - acc_top1: 0.7142 - acc_top5: 0.9348 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.2138 - acc_top1: 0.3054 - acc_top5: 0.5843 - 30ms/step
Eval samples: 10000
Epoch 90/100
step 391/391 [] - loss: 1.0687 - acc_top1: 0.7176 - acc_top5: 0.9373 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9641 - acc_top1: 0.3103 - acc_top5: 0.5898 - 29ms/step
Eval samples: 10000
Epoch 91/100
step 391/391 [] - loss: 1.4099 - acc_top1: 0.7216 - acc_top5: 0.9381 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.3445 - acc_top1: 0.3144 - acc_top5: 0.5849 - 29ms/step
Eval samples: 10000
Epoch 92/100
step 391/391 [] - loss: 0.9097 - acc_top1: 0.7285 - acc_top5: 0.9405 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.7660 - acc_top1: 0.3144 - acc_top5: 0.5947 - 30ms/step
Eval samples: 10000
Epoch 93/100
step 391/391 [] - loss: 0.7941 - acc_top1: 0.7298 - acc_top5: 0.9421 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.3507 - acc_top1: 0.3180 - acc_top5: 0.5912 - 30ms/step
Eval samples: 10000
Epoch 94/100
step 391/391 [] - loss: 1.2478 - acc_top1: 0.7398 - acc_top5: 0.9460 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.2983 - acc_top1: 0.3197 - acc_top5: 0.5926 - 30ms/step
Eval samples: 10000
Epoch 95/100
step 391/391 [] - loss: 0.9032 - acc_top1: 0.7401 - acc_top5: 0.9485 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.3697 - acc_top1: 0.3224 - acc_top5: 0.5991 - 29ms/step
Eval samples: 10000
Epoch 96/100
step 391/391 [] - loss: 0.8351 - acc_top1: 0.7491 - acc_top5: 0.9488 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.9680 - acc_top1: 0.3210 - acc_top5: 0.5995 - 29ms/step
Eval samples: 10000
Epoch 97/100
step 391/391 [] - loss: 1.0612 - acc_top1: 0.7601 - acc_top5: 0.9523 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.2625 - acc_top1: 0.3166 - acc_top5: 0.5961 - 37ms/step
Eval samples: 10000
Epoch 98/100
step 391/391 [] - loss: 0.8611 - acc_top1: 0.7618 - acc_top5: 0.9553 - 51ms/step
Eval begin…
step 79/79 [] - loss: 2.8326 - acc_top1: 0.3220 - acc_top5: 0.6042 - 30ms/step
Eval samples: 10000
Epoch 99/100
step 391/391 [] - loss: 1.0832 - acc_top1: 0.7693 - acc_top5: 0.9560 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.1979 - acc_top1: 0.3192 - acc_top5: 0.6012 - 32ms/step
Eval samples: 10000
Epoch 100/100
step 391/391 [] - loss: 0.9606 - acc_top1: 0.7716 - acc_top5: 0.9584 - 51ms/step
Eval begin…
step 79/79 [] - loss: 3.2596 - acc_top1: 0.3246 - acc_top5: 0.6043 - 29ms/step
Eval samples: 10000
四、基于ResNet50-NAM意力机制的cifar100实验
4.1 导入NAM注意力机制
In [17]
import paddle.nn as nn
import paddle
from paddle.nn import functional as F

class Channel_Att(nn.Layer):
def init(self, channels=3, t=16):
super(Channel_Att, self).init()
self.channels = channels

    self.bn2 = nn.BatchNorm2D(self.channels)


def forward(self, x):
    residual = x

    x = self.bn2(x)

    weight_bn = self.bn2.weight.abs() / paddle.sum(self.bn2.weight.abs())

    x = x.transpose([0, 2, 3, 1])
    x = paddle.multiply(weight_bn, x)
    x = x.transpose([0, 3, 1, 2])
    
    x = F.sigmoid(x) * residual #
    
    return x

class Att(nn.Layer):
def init(self, channels=3, out_channels=None, no_spatial=True):
super(Att, self).init()
self.Channel_Att = Channel_Att(channels)

def forward(self, x):
    x_out1=self.Channel_Att(x)

    return x_out1

4.2.搭建ResNet-NAM网络模型
此网络模型模型同时适用ResNet18、ResNet34、ResNet50、ResNet101、ResNet152

In [18]
all = []
model_urls = {
‘resnet18’: (‘https://paddle-hapi.bj.bcebos.com/models/resnet18.pdparams’,
‘cf548f46534aa3560945be4b95cd11c4’),
‘resnet34’: (‘https://paddle-hapi.bj.bcebos.com/models/resnet34.pdparams’,
‘8d2275cf8706028345f78ac0e1d31969’),
‘resnet50’: (‘https://paddle-hapi.bj.bcebos.com/models/resnet50.pdparams’,
‘ca6f485ee1ab0492d38f323885b0ad80’),
‘resnet101’: (‘https://paddle-hapi.bj.bcebos.com/models/resnet101.pdparams’,
‘02f35f034ca3858e1e54d4036443c92d’),
‘resnet152’: (‘https://paddle-hapi.bj.bcebos.com/models/resnet152.pdparams’,
‘7ad16a2f1e7333859ff986138630fd7a’),
}

class BasicBlock(nn.Layer):
expansion = 1

def __init__(self,
             inplanes,
             planes,
             stride=1,
             downsample=None,
             groups=1,
             base_width=64,
             dilation=1,
             norm_layer=None):
    super(BasicBlock, self).__init__()
    if norm_layer is None:
        norm_layer = nn.BatchNorm2D

    if dilation > 1:
        raise NotImplementedError(
            "Dilation > 1 not supported in BasicBlock")

    self.conv1 = nn.Conv2D(
        inplanes, planes, 3, padding=1, stride=stride, bias_attr=False)
    self.bn1 = norm_layer(planes)
    self.relu = nn.ReLU()
    self.conv2 = nn.Conv2D(planes, planes, 3, padding=1, bias_attr=False)
    self.bn2 = norm_layer(planes)
    self.downsample = downsample
    self.stride = stride
    self.nam = Att(planes)

def forward(self, x):
    identity = x
    out = self.conv1(x)
    out = self.bn1(out)
    out = self.relu(out)
    out = self.conv2(out)
    out = self.bn2(out)
    if self.downsample is not None:
        identity = self.downsample(x)
    out = self.nam(out)
    out += identity
    out = self.relu(out)

    return out

class BottleneckBlock(nn.Layer):

expansion = 4

def __init__(self,
             inplanes,
             planes,
             stride=1,
             downsample=None,
             groups=1,
             base_width=64,
             dilation=1,
             norm_layer=None):
    super(BottleneckBlock, self).__init__()
    if norm_layer is None:
        norm_layer = nn.BatchNorm2D
    width = int(planes * (base_width / 64.)) * groups
    self.conv1 = nn.Conv2D(inplanes, width, 1, bias_attr=False)
    self.bn1 = norm_layer(width)
    self.conv2 = nn.Conv2D(
        width,
        width,
        3,
        padding=dilation,
        stride=stride,
        groups=groups,
        dilation=dilation,
        bias_attr=False)
    self.bn2 = norm_layer(width)
    self.conv3 = nn.Conv2D(
        width, planes * self.expansion, 1, bias_attr=False)
    self.bn3 = norm_layer(planes * self.expansion)
    self.relu = nn.ReLU()
    self.downsample = downsample
    self.stride = stride
    self.nam = Att(planes*4)

def forward(self, x):
    identity = x

    out = self.conv1(x)
    out = self.bn1(out)
    out = self.relu(out)

    out = self.conv2(out)
    out = self.bn2(out)
    out = self.relu(out)

    out = self.conv3(out)
    out = self.bn3(out)

    if self.downsample is not None:
        identity = self.downsample(x)
    out = self.nam(out)
    out += identity
    out = self.relu(out)

    return out

class ResNet(nn.Layer):
“”"ResNet model from
"Deep Residual Learning for Image Recognition" <https://arxiv.org/pdf/1512.03385.pdf>_

Args:
    Block (BasicBlock|BottleneckBlock): block module of model.
    depth (int): layers of resnet, default: 50.
    num_classes (int): output dim of last fc layer. If num_classes <=0, last fc layer
                        will not be defined. Default: 1000.
    with_pool (bool): use pool before the last fc layer or not. Default: True.

Examples:
    .. code-block:: python

        from paddle.vision.models import ResNet
        from paddle.vision.models.resnet import BottleneckBlock, BasicBlock

        resnet50 = ResNet(BottleneckBlock, 50)

        resnet18 = ResNet(BasicBlock, 18)

"""

def __init__(self, block, depth, num_classes=100, with_pool=True):
    super(ResNet, self).__init__()
    layer_cfg = {
        18: [2, 2, 2, 2],
        34: [3, 4, 6, 3],
        50: [3, 4, 6, 3],
        101: [3, 4, 23, 3],
        152: [3, 8, 36, 3]
    }
    layers = layer_cfg[depth]
    self.num_classes = num_classes
    self.with_pool = with_pool
    self._norm_layer = nn.BatchNorm2D

    self.inplanes = 64
    self.dilation = 1
    
    ###
    # 将大核卷积改为小核卷积
    ###

    self.conv1 = nn.Conv2D(
        3,
        self.inplanes,
        kernel_size=3,
        stride=1,
        padding=1,
        bias_attr=False)
    self.bn1 = self._norm_layer(self.inplanes)
    self.relu = nn.ReLU()
    self.layer1 = self._make_layer(block, 64, layers[0])
    self.layer2 = self._make_layer(block, 128, layers[1], stride=2)
    self.layer3 = self._make_layer(block, 256, layers[2], stride=2)
    self.layer4 = self._make_layer(block, 512, layers[3], stride=2)
    if with_pool:
        self.avgpool = nn.AdaptiveAvgPool2D((1, 1))

    if num_classes > 0:
        self.fc = nn.Linear(512 * block.expansion, num_classes)

def _make_layer(self, block, planes, blocks, stride=1, dilate=False):
    norm_layer = self._norm_layer
    downsample = None
    previous_dilation = self.dilation
    if dilate:
        self.dilation *= stride
        stride = 1
    if stride != 1 or self.inplanes != planes * block.expansion:
        downsample = nn.Sequential(
            nn.Conv2D(
                self.inplanes,
                planes * block.expansion,
                1,
                stride=stride,
                bias_attr=False),
            norm_layer(planes * block.expansion), )

    layers = []
    layers.append(
        block(self.inplanes, planes, stride, downsample, 1, 64,
              previous_dilation, norm_layer))
    self.inplanes = planes * block.expansion
    for _ in range(1, blocks):
        layers.append(block(self.inplanes, planes, norm_layer=norm_layer))

    return nn.Sequential(*layers)

def forward(self, x):
    x = self.conv1(x)
    x = self.bn1(x)
    x = self.relu(x)

    ### 
    # 去掉池化
    ###
    
    # x = self.maxpool(x)
    x = self.layer1(x)
    x = self.layer2(x)
    x = self.layer3(x)
    x = self.layer4(x)

    if self.with_pool:
        x = self.avgpool(x)

    if self.num_classes > 0:
        x = paddle.flatten(x, 1)
        x = self.fc(x)

    return x

def _resnet(arch, Block, depth, pretrained, **kwargs):
model = ResNet(Block, depth, **kwargs)
if pretrained:
assert arch in model_urls, “{} model do not have a pretrained model now, you should set pretrained=False”.format(
arch)
weight_path = get_weights_path_from_url(model_urls[arch][0],
model_urls[arch][1])

    param = paddle.load(weight_path)
    model.set_dict(param)

return model

def resnet50(pretrained=False, **kwargs):
“”"ResNet 50-layer model

Args:
    pretrained (bool): If True, returns a model pre-trained on ImageNet

Examples:
    .. code-block:: python

        from paddle.vision.models import resnet50

        # build model
        model = resnet50()

        # build model and load imagenet pretrained weight
        # model = resnet50(pretrained=True)
"""
return _resnet('resnet50', BottleneckBlock, 50, pretrained, **kwargs)

def resnet18(pretrained=False, **kwargs):

return _resnet('resnet18', BasicBlock, 18, pretrained, **kwargs)

4.3.实例化网络模型
使用summary可视化模型参数

In [19]
resnet = resnet50()
model = paddle.Model(resnet)

#模型可视化
model.summary((-1, 3, 32, 32))

Layer (type) Input Shape Output Shape Param #

Conv2D-165        [[1, 3, 32, 32]]     [1, 64, 32, 32]         1,728

BatchNorm2D-165 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-54 [[1, 64, 32, 32]] [1, 64, 32, 32] 0
Conv2D-167 [[1, 64, 32, 32]] [1, 64, 32, 32] 4,096
BatchNorm2D-167 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-55 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
Conv2D-168 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,864
BatchNorm2D-168 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
Conv2D-169 [[1, 64, 32, 32]] [1, 256, 32, 32] 16,384
BatchNorm2D-169 [[1, 256, 32, 32]] [1, 256, 32, 32] 1,024
Conv2D-166 [[1, 64, 32, 32]] [1, 256, 32, 32] 16,384
BatchNorm2D-166 [[1, 256, 32, 32]] [1, 256, 32, 32] 1,024
BatchNorm2D-170 [[1, 256, 32, 32]] [1, 256, 32, 32] 1,024
Channel_Att-1 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
Att-1 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
BottleneckBlock-50 [[1, 64, 32, 32]] [1, 256, 32, 32] 0
Conv2D-170 [[1, 256, 32, 32]] [1, 64, 32, 32] 16,384
BatchNorm2D-171 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-56 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
Conv2D-171 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,864
BatchNorm2D-172 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
Conv2D-172 [[1, 64, 32, 32]] [1, 256, 32, 32] 16,384
BatchNorm2D-173 [[1, 256, 32, 32]] [1, 256, 32, 32] 1,024
BatchNorm2D-174 [[1, 256, 32, 32]] [1, 256, 32, 32] 1,024
Channel_Att-2 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
Att-2 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
BottleneckBlock-51 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
Conv2D-173 [[1, 256, 32, 32]] [1, 64, 32, 32] 16,384
BatchNorm2D-175 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
ReLU-57 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
Conv2D-174 [[1, 64, 32, 32]] [1, 64, 32, 32] 36,864
BatchNorm2D-176 [[1, 64, 32, 32]] [1, 64, 32, 32] 256
Conv2D-175 [[1, 64, 32, 32]] [1, 256, 32, 32] 16,384
BatchNorm2D-177 [[1, 256, 32, 32]] [1, 256, 32, 32] 1,024
BatchNorm2D-178 [[1, 256, 32, 32]] [1, 256, 32, 32] 1,024
Channel_Att-3 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
Att-3 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
BottleneckBlock-52 [[1, 256, 32, 32]] [1, 256, 32, 32] 0
Conv2D-177 [[1, 256, 32, 32]] [1, 128, 32, 32] 32,768
BatchNorm2D-180 [[1, 128, 32, 32]] [1, 128, 32, 32] 512
ReLU-58 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Conv2D-178 [[1, 128, 32, 32]] [1, 128, 16, 16] 147,456
BatchNorm2D-181 [[1, 128, 16, 16]] [1, 128, 16, 16] 512
Conv2D-179 [[1, 128, 16, 16]] [1, 512, 16, 16] 65,536
BatchNorm2D-182 [[1, 512, 16, 16]] [1, 512, 16, 16] 2,048
Conv2D-176 [[1, 256, 32, 32]] [1, 512, 16, 16] 131,072
BatchNorm2D-179 [[1, 512, 16, 16]] [1, 512, 16, 16] 2,048
BatchNorm2D-183 [[1, 512, 16, 16]] [1, 512, 16, 16] 2,048
Channel_Att-4 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Att-4 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
BottleneckBlock-53 [[1, 256, 32, 32]] [1, 512, 16, 16] 0
Conv2D-180 [[1, 512, 16, 16]] [1, 128, 16, 16] 65,536
BatchNorm2D-184 [[1, 128, 16, 16]] [1, 128, 16, 16] 512
ReLU-59 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Conv2D-181 [[1, 128, 16, 16]] [1, 128, 16, 16] 147,456
BatchNorm2D-185 [[1, 128, 16, 16]] [1, 128, 16, 16] 512
Conv2D-182 [[1, 128, 16, 16]] [1, 512, 16, 16] 65,536
BatchNorm2D-186 [[1, 512, 16, 16]] [1, 512, 16, 16] 2,048
BatchNorm2D-187 [[1, 512, 16, 16]] [1, 512, 16, 16] 2,048
Channel_Att-5 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Att-5 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
BottleneckBlock-54 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Conv2D-183 [[1, 512, 16, 16]] [1, 128, 16, 16] 65,536
BatchNorm2D-188 [[1, 128, 16, 16]] [1, 128, 16, 16] 512
ReLU-60 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Conv2D-184 [[1, 128, 16, 16]] [1, 128, 16, 16] 147,456
BatchNorm2D-189 [[1, 128, 16, 16]] [1, 128, 16, 16] 512
Conv2D-185 [[1, 128, 16, 16]] [1, 512, 16, 16] 65,536
BatchNorm2D-190 [[1, 512, 16, 16]] [1, 512, 16, 16] 2,048
BatchNorm2D-191 [[1, 512, 16, 16]] [1, 512, 16, 16] 2,048
Channel_Att-6 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Att-6 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
BottleneckBlock-55 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Conv2D-186 [[1, 512, 16, 16]] [1, 128, 16, 16] 65,536
BatchNorm2D-192 [[1, 128, 16, 16]] [1, 128, 16, 16] 512
ReLU-61 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Conv2D-187 [[1, 128, 16, 16]] [1, 128, 16, 16] 147,456
BatchNorm2D-193 [[1, 128, 16, 16]] [1, 128, 16, 16] 512
Conv2D-188 [[1, 128, 16, 16]] [1, 512, 16, 16] 65,536
BatchNorm2D-194 [[1, 512, 16, 16]] [1, 512, 16, 16] 2,048
BatchNorm2D-195 [[1, 512, 16, 16]] [1, 512, 16, 16] 2,048
Channel_Att-7 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Att-7 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
BottleneckBlock-56 [[1, 512, 16, 16]] [1, 512, 16, 16] 0
Conv2D-190 [[1, 512, 16, 16]] [1, 256, 16, 16] 131,072
BatchNorm2D-197 [[1, 256, 16, 16]] [1, 256, 16, 16] 1,024
ReLU-62 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-191 [[1, 256, 16, 16]] [1, 256, 8, 8] 589,824
BatchNorm2D-198 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
Conv2D-192 [[1, 256, 8, 8]] [1, 1024, 8, 8] 262,144
BatchNorm2D-199 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
Conv2D-189 [[1, 512, 16, 16]] [1, 1024, 8, 8] 524,288
BatchNorm2D-196 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
BatchNorm2D-200 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
Channel_Att-8 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Att-8 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
BottleneckBlock-57 [[1, 512, 16, 16]] [1, 1024, 8, 8] 0
Conv2D-193 [[1, 1024, 8, 8]] [1, 256, 8, 8] 262,144
BatchNorm2D-201 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
ReLU-63 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-194 [[1, 256, 8, 8]] [1, 256, 8, 8] 589,824
BatchNorm2D-202 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
Conv2D-195 [[1, 256, 8, 8]] [1, 1024, 8, 8] 262,144
BatchNorm2D-203 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
BatchNorm2D-204 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
Channel_Att-9 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Att-9 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
BottleneckBlock-58 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-196 [[1, 1024, 8, 8]] [1, 256, 8, 8] 262,144
BatchNorm2D-205 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
ReLU-64 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-197 [[1, 256, 8, 8]] [1, 256, 8, 8] 589,824
BatchNorm2D-206 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
Conv2D-198 [[1, 256, 8, 8]] [1, 1024, 8, 8] 262,144
BatchNorm2D-207 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
BatchNorm2D-208 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
Channel_Att-10 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Att-10 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
BottleneckBlock-59 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-199 [[1, 1024, 8, 8]] [1, 256, 8, 8] 262,144
BatchNorm2D-209 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
ReLU-65 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-200 [[1, 256, 8, 8]] [1, 256, 8, 8] 589,824
BatchNorm2D-210 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
Conv2D-201 [[1, 256, 8, 8]] [1, 1024, 8, 8] 262,144
BatchNorm2D-211 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
BatchNorm2D-212 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
Channel_Att-11 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Att-11 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
BottleneckBlock-60 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-202 [[1, 1024, 8, 8]] [1, 256, 8, 8] 262,144
BatchNorm2D-213 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
ReLU-66 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-203 [[1, 256, 8, 8]] [1, 256, 8, 8] 589,824
BatchNorm2D-214 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
Conv2D-204 [[1, 256, 8, 8]] [1, 1024, 8, 8] 262,144
BatchNorm2D-215 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
BatchNorm2D-216 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
Channel_Att-12 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Att-12 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
BottleneckBlock-61 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-205 [[1, 1024, 8, 8]] [1, 256, 8, 8] 262,144
BatchNorm2D-217 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
ReLU-67 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-206 [[1, 256, 8, 8]] [1, 256, 8, 8] 589,824
BatchNorm2D-218 [[1, 256, 8, 8]] [1, 256, 8, 8] 1,024
Conv2D-207 [[1, 256, 8, 8]] [1, 1024, 8, 8] 262,144
BatchNorm2D-219 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
BatchNorm2D-220 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 4,096
Channel_Att-13 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Att-13 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
BottleneckBlock-62 [[1, 1024, 8, 8]] [1, 1024, 8, 8] 0
Conv2D-209 [[1, 1024, 8, 8]] [1, 512, 8, 8] 524,288
BatchNorm2D-222 [[1, 512, 8, 8]] [1, 512, 8, 8] 2,048
ReLU-68 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
Conv2D-210 [[1, 512, 8, 8]] [1, 512, 4, 4] 2,359,296
BatchNorm2D-223 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,048
Conv2D-211 [[1, 512, 4, 4]] [1, 2048, 4, 4] 1,048,576
BatchNorm2D-224 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 8,192
Conv2D-208 [[1, 1024, 8, 8]] [1, 2048, 4, 4] 2,097,152
BatchNorm2D-221 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 8,192
BatchNorm2D-225 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 8,192
Channel_Att-14 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
Att-14 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
BottleneckBlock-63 [[1, 1024, 8, 8]] [1, 2048, 4, 4] 0
Conv2D-212 [[1, 2048, 4, 4]] [1, 512, 4, 4] 1,048,576
BatchNorm2D-226 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,048
ReLU-69 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
Conv2D-213 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,359,296
BatchNorm2D-227 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,048
Conv2D-214 [[1, 512, 4, 4]] [1, 2048, 4, 4] 1,048,576
BatchNorm2D-228 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 8,192
BatchNorm2D-229 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 8,192
Channel_Att-15 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
Att-15 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
BottleneckBlock-64 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
Conv2D-215 [[1, 2048, 4, 4]] [1, 512, 4, 4] 1,048,576
BatchNorm2D-230 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,048
ReLU-70 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
Conv2D-216 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,359,296
BatchNorm2D-231 [[1, 512, 4, 4]] [1, 512, 4, 4] 2,048
Conv2D-217 [[1, 512, 4, 4]] [1, 2048, 4, 4] 1,048,576
BatchNorm2D-232 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 8,192
BatchNorm2D-233 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 8,192
Channel_Att-16 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
Att-16 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
BottleneckBlock-65 [[1, 2048, 4, 4]] [1, 2048, 4, 4] 0
AdaptiveAvgPool2D-4 [[1, 2048, 4, 4]] [1, 2048, 1, 1] 0
Linear-4 [[1, 2048]] [1, 100] 204,900

Total params: 23,818,788
Trainable params: 23,652,132
Non-trainable params: 166,656

Input size (MB): 0.01
Forward/backward pass size (MB): 121.64
Params size (MB): 90.86
Estimated Total Size (MB): 212.51

{‘total_params’: 23818788, ‘trainable_params’: 23652132}
4.4模型训练
我们使用momentum这个动量优化函数，交叉熵损失函数，训练轮数为100

In [24]
from paddle.optimizer.lr import CosineAnnealingDecay, MultiStepDecay, LinearWarmup

#开始模型训练
model.fit(train_dataset,
eval_dataset,
epochs=100,#训练的轮数
batch_size=128,#每次训练多少个
verbose=1,#显示模式
shuffle=True,#打乱数据集顺序
num_workers=4,
callbacks=callback_visualdl,
)

callback_visualdl = paddle.callbacks.VisualDL(log_dir=‘visualdl_log_dir-NAM’)
The loss value printed in the log is the current step, and the metric is the average value of previous steps.
Epoch 1/100
step 391/391 [] - loss: 4.8497 - acc_top1: 0.0111 - acc_top5: 0.0573 - 88ms/step
Eval begin…
step 79/79 [] - loss: 4.8184 - acc_top1: 0.0137 - acc_top5: 0.0692 - 46ms/step
Eval samples: 10000
Epoch 2/100
step 391/391 [] - loss: 4.7389 - acc_top1: 0.0179 - acc_top5: 0.0762 - 86ms/step
Eval begin…
step 79/79 [] - loss: 4.5421 - acc_top1: 0.0212 - acc_top5: 0.0901 - 48ms/step
Eval samples: 10000
Epoch 3/100
step 391/391 [] - loss: 4.5462 - acc_top1: 0.0318 - acc_top5: 0.1244 - 91ms/step
Eval begin…
step 79/79 [] - loss: 4.5771 - acc_top1: 0.0473 - acc_top5: 0.1673 - 45ms/step
Eval samples: 10000
Epoch 4/100
step 391/391 [] - loss: 4.0376 - acc_top1: 0.0583 - acc_top5: 0.2006 - 87ms/step
Eval begin…
step 79/79 [] - loss: 4.2838 - acc_top1: 0.0760 - acc_top5: 0.2449 - 47ms/step
Eval samples: 10000
Epoch 5/100
step 391/391 [] - loss: 3.9339 - acc_top1: 0.0872 - acc_top5: 0.2677 - 87ms/step
Eval begin…
step 79/79 [] - loss: 4.0610 - acc_top1: 0.1020 - acc_top5: 0.2914 - 46ms/step
Eval samples: 10000
Epoch 6/100
step 391/391 [] - loss: 3.5850 - acc_top1: 0.1119 - acc_top5: 0.3213 - 94ms/step
Eval begin…
step 79/79 [] - loss: 3.9599 - acc_top1: 0.1227 - acc_top5: 0.3317 - 51ms/step
Eval samples: 10000
Epoch 7/100
step 391/391 [] - loss: 3.7031 - acc_top1: 0.1271 - acc_top5: 0.3470 - 89ms/step
Eval begin…
step 79/79 [] - loss: 3.9104 - acc_top1: 0.1420 - acc_top5: 0.3642 - 46ms/step
Eval samples: 10000
Epoch 8/100
step 391/391 [] - loss: 3.6172 - acc_top1: 0.1409 - acc_top5: 0.3728 - 86ms/step
Eval begin…
step 79/79 [] - loss: 3.7252 - acc_top1: 0.1502 - acc_top5: 0.3806 - 45ms/step
Eval samples: 10000
Epoch 9/100
step 391/391 [] - loss: 3.5688 - acc_top1: 0.1514 - acc_top5: 0.3920 - 100ms/step
Eval begin…
step 79/79 [] - loss: 3.6717 - acc_top1: 0.1567 - acc_top5: 0.3929 - 46ms/step
Eval samples: 10000
Epoch 10/100
step 391/391 [] - loss: 3.6321 - acc_top1: 0.1633 - acc_top5: 0.4103 - 86ms/step
Eval begin…
step 79/79 [] - loss: 3.6300 - acc_top1: 0.1653 - acc_top5: 0.4026 - 48ms/step
Eval samples: 10000
Epoch 11/100
step 391/391 [] - loss: 3.5487 - acc_top1: 0.1704 - acc_top5: 0.4237 - 89ms/step
Eval begin…
step 79/79 [] - loss: 3.7029 - acc_top1: 0.1760 - acc_top5: 0.4197 - 46ms/step
Eval samples: 10000
Epoch 12/100
step 391/391 [] - loss: 3.8424 - acc_top1: 0.1797 - acc_top5: 0.4357 - 87ms/step
Eval begin…
step 79/79 [] - loss: 3.4210 - acc_top1: 0.1766 - acc_top5: 0.4333 - 48ms/step
Eval samples: 10000
Epoch 13/100
step 391/391 [] - loss: 3.6314 - acc_top1: 0.1887 - acc_top5: 0.4541 - 89ms/step
Eval begin…
step 79/79 [] - loss: 3.4551 - acc_top1: 0.1874 - acc_top5: 0.4394 - 49ms/step
Eval samples: 10000
Epoch 14/100
step 391/391 [] - loss: 2.9399 - acc_top1: 0.1986 - acc_top5: 0.4647 - 88ms/step
Eval begin…
step 79/79 [] - loss: 3.4774 - acc_top1: 0.1918 - acc_top5: 0.4490 - 47ms/step
Eval samples: 10000
Epoch 15/100
step 391/391 [] - loss: 3.5771 - acc_top1: 0.2082 - acc_top5: 0.4766 - 88ms/step
Eval begin…
step 79/79 [] - loss: 3.2711 - acc_top1: 0.1947 - acc_top5: 0.4522 - 46ms/step
Eval samples: 10000
Epoch 16/100
step 391/391 [] - loss: 3.3304 - acc_top1: 0.2145 - acc_top5: 0.4872 - 86ms/step
Eval begin…
step 79/79 [] - loss: 3.1900 - acc_top1: 0.2015 - acc_top5: 0.4576 - 45ms/step
Eval samples: 10000
Epoch 17/100
step 391/391 [] - loss: 3.5043 - acc_top1: 0.2237 - acc_top5: 0.4994 - 88ms/step
Eval begin…
step 79/79 [] - loss: 3.4247 - acc_top1: 0.1992 - acc_top5: 0.4625 - 45ms/step
Eval samples: 10000
Epoch 18/100
step 391/391 [] - loss: 3.3019 - acc_top1: 0.2264 - acc_top5: 0.5060 - 87ms/step
Eval begin…
step 79/79 [] - loss: 3.2735 - acc_top1: 0.2071 - acc_top5: 0.4673 - 45ms/step
Eval samples: 10000
Epoch 19/100
step 391/391 [] - loss: 3.1922 - acc_top1: 0.2345 - acc_top5: 0.5142 - 91ms/step
Eval begin…
step 79/79 [] - loss: 3.2611 - acc_top1: 0.2136 - acc_top5: 0.4784 - 46ms/step
Eval samples: 10000
Epoch 20/100
step 391/391 [] - loss: 2.9049 - acc_top1: 0.2388 - acc_top5: 0.5234 - 89ms/step
Eval begin…
step 79/79 [] - loss: 3.2051 - acc_top1: 0.2146 - acc_top5: 0.4867 - 45ms/step
Eval samples: 10000
Epoch 21/100
step 391/391 [] - loss: 2.9065 - acc_top1: 0.2464 - acc_top5: 0.5364 - 91ms/step
Eval begin…
step 79/79 [] - loss: 3.3232 - acc_top1: 0.2204 - acc_top5: 0.4860 - 45ms/step
Eval samples: 10000
Epoch 22/100
step 391/391 [] - loss: 2.8180 - acc_top1: 0.2546 - acc_top5: 0.5443 - 88ms/step
Eval begin…
step 79/79 [] - loss: 3.0776 - acc_top1: 0.2184 - acc_top5: 0.4927 - 46ms/step
Eval samples: 10000
Epoch 23/100
step 391/391 [] - loss: 3.3415 - acc_top1: 0.2574 - acc_top5: 0.5520 - 87ms/step
Eval begin…
step 79/79 [] - loss: 3.1972 - acc_top1: 0.2295 - acc_top5: 0.4959 - 45ms/step
Eval samples: 10000
Epoch 24/100
step 391/391 [] - loss: 3.1298 - acc_top1: 0.2646 - acc_top5: 0.5576 - 87ms/step
Eval begin…
step 79/79 [] - loss: 3.0983 - acc_top1: 0.2341 - acc_top5: 0.5087 - 45ms/step
Eval samples: 10000
Epoch 25/100
step 391/391 [] - loss: 2.9851 - acc_top1: 0.2744 - acc_top5: 0.5661 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.9562 - acc_top1: 0.2353 - acc_top5: 0.5048 - 45ms/step
Eval samples: 10000
Epoch 26/100
step 391/391 [] - loss: 2.9751 - acc_top1: 0.2788 - acc_top5: 0.5743 - 87ms/step
Eval begin…
step 79/79 [] - loss: 3.1786 - acc_top1: 0.2448 - acc_top5: 0.5172 - 46ms/step
Eval samples: 10000
Epoch 27/100
step 391/391 [] - loss: 3.0608 - acc_top1: 0.2819 - acc_top5: 0.5805 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.9408 - acc_top1: 0.2470 - acc_top5: 0.5202 - 45ms/step
Eval samples: 10000
Epoch 28/100
step 391/391 [] - loss: 3.0520 - acc_top1: 0.2899 - acc_top5: 0.5886 - 88ms/step
Eval begin…
step 79/79 [] - loss: 3.1723 - acc_top1: 0.2506 - acc_top5: 0.5260 - 46ms/step
Eval samples: 10000
Epoch 29/100
step 391/391 [] - loss: 2.7072 - acc_top1: 0.2943 - acc_top5: 0.5966 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.9665 - acc_top1: 0.2507 - acc_top5: 0.5346 - 45ms/step
Eval samples: 10000
Epoch 30/100
step 391/391 [] - loss: 2.9159 - acc_top1: 0.3018 - acc_top5: 0.6030 - 86ms/step
Eval begin…
step 79/79 [] - loss: 2.9954 - acc_top1: 0.2521 - acc_top5: 0.5332 - 45ms/step
Eval samples: 10000
Epoch 31/100
step 391/391 [] - loss: 2.6270 - acc_top1: 0.3087 - acc_top5: 0.6109 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.8555 - acc_top1: 0.2601 - acc_top5: 0.5434 - 46ms/step
Eval samples: 10000
Epoch 32/100
step 391/391 [] - loss: 3.0665 - acc_top1: 0.3113 - acc_top5: 0.6158 - 94ms/step
Eval begin…
step 79/79 [] - loss: 3.0362 - acc_top1: 0.2614 - acc_top5: 0.5450 - 45ms/step
Eval samples: 10000
Epoch 33/100
step 391/391 [] - loss: 3.3042 - acc_top1: 0.3189 - acc_top5: 0.6238 - 94ms/step
Eval begin…
step 79/79 [] - loss: 2.9307 - acc_top1: 0.2604 - acc_top5: 0.5492 - 45ms/step
Eval samples: 10000
Epoch 34/100
step 391/391 [] - loss: 2.6142 - acc_top1: 0.3244 - acc_top5: 0.6288 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.9296 - acc_top1: 0.2680 - acc_top5: 0.5458 - 45ms/step
Eval samples: 10000
Epoch 35/100
step 391/391 [] - loss: 2.8182 - acc_top1: 0.3319 - acc_top5: 0.6381 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.8653 - acc_top1: 0.2727 - acc_top5: 0.5506 - 45ms/step
Eval samples: 10000
Epoch 36/100
step 391/391 [] - loss: 2.6769 - acc_top1: 0.3386 - acc_top5: 0.6423 - 87ms/step
Eval begin…
step 79/79 [] - loss: 3.1202 - acc_top1: 0.2668 - acc_top5: 0.5518 - 46ms/step
Eval samples: 10000
Epoch 37/100
step 391/391 [] - loss: 2.3541 - acc_top1: 0.3409 - acc_top5: 0.6498 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.7944 - acc_top1: 0.2655 - acc_top5: 0.5587 - 46ms/step
Eval samples: 10000
Epoch 38/100
step 391/391 [] - loss: 2.6344 - acc_top1: 0.3489 - acc_top5: 0.6554 - 93ms/step
Eval begin…
step 79/79 [] - loss: 3.1045 - acc_top1: 0.2676 - acc_top5: 0.5529 - 46ms/step
Eval samples: 10000
Epoch 39/100
step 391/391 [] - loss: 2.4571 - acc_top1: 0.3537 - acc_top5: 0.6611 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.6990 - acc_top1: 0.2730 - acc_top5: 0.5585 - 46ms/step
Eval samples: 10000
Epoch 40/100
step 391/391 [] - loss: 2.5621 - acc_top1: 0.3624 - acc_top5: 0.6693 - 86ms/step
Eval begin…
step 79/79 [] - loss: 2.7441 - acc_top1: 0.2717 - acc_top5: 0.5591 - 46ms/step
Eval samples: 10000
Epoch 41/100
step 391/391 [] - loss: 2.6501 - acc_top1: 0.3660 - acc_top5: 0.6726 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.9928 - acc_top1: 0.2753 - acc_top5: 0.5627 - 49ms/step
Eval samples: 10000
Epoch 42/100
step 391/391 [] - loss: 2.3623 - acc_top1: 0.3721 - acc_top5: 0.6811 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.7311 - acc_top1: 0.2707 - acc_top5: 0.5699 - 45ms/step
Eval samples: 10000
Epoch 43/100
step 391/391 [] - loss: 2.4758 - acc_top1: 0.3746 - acc_top5: 0.6850 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.7034 - acc_top1: 0.2764 - acc_top5: 0.5635 - 46ms/step
Eval samples: 10000
Epoch 44/100
step 391/391 [] - loss: 2.5367 - acc_top1: 0.3816 - acc_top5: 0.6903 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.9082 - acc_top1: 0.2750 - acc_top5: 0.5644 - 46ms/step
Eval samples: 10000
Epoch 45/100
step 391/391 [] - loss: 2.5759 - acc_top1: 0.3872 - acc_top5: 0.6998 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.7683 - acc_top1: 0.2749 - acc_top5: 0.5676 - 46ms/step
Eval samples: 10000
Epoch 46/100
step 391/391 [] - loss: 2.5877 - acc_top1: 0.3934 - acc_top5: 0.7032 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.7031 - acc_top1: 0.2860 - acc_top5: 0.5820 - 46ms/step
Eval samples: 10000
Epoch 47/100
step 391/391 [] - loss: 2.4155 - acc_top1: 0.3974 - acc_top5: 0.7081 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.7535 - acc_top1: 0.2755 - acc_top5: 0.5733 - 51ms/step
Eval samples: 10000
Epoch 48/100
step 391/391 [] - loss: 2.5510 - acc_top1: 0.4023 - acc_top5: 0.7103 - 87ms/step
Eval begin…
step 79/79 [] - loss: 3.0216 - acc_top1: 0.2804 - acc_top5: 0.5679 - 46ms/step
Eval samples: 10000
Epoch 49/100
step 391/391 [] - loss: 2.5306 - acc_top1: 0.4058 - acc_top5: 0.7172 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.7141 - acc_top1: 0.2944 - acc_top5: 0.5844 - 45ms/step
Eval samples: 10000
Epoch 50/100
step 391/391 [] - loss: 2.2112 - acc_top1: 0.4125 - acc_top5: 0.7234 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.7876 - acc_top1: 0.2884 - acc_top5: 0.5834 - 46ms/step
Eval samples: 10000
Epoch 51/100
step 391/391 [] - loss: 2.3451 - acc_top1: 0.4178 - acc_top5: 0.7263 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.8956 - acc_top1: 0.3044 - acc_top5: 0.5890 - 46ms/step
Eval samples: 10000
Epoch 52/100
step 391/391 [] - loss: 2.2558 - acc_top1: 0.4235 - acc_top5: 0.7319 - 95ms/step
Eval begin…
step 79/79 [] - loss: 2.9233 - acc_top1: 0.2966 - acc_top5: 0.5875 - 47ms/step
Eval samples: 10000
Epoch 53/100
step 391/391 [] - loss: 2.5057 - acc_top1: 0.4301 - acc_top5: 0.7376 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.5221 - acc_top1: 0.3001 - acc_top5: 0.5928 - 45ms/step
Eval samples: 10000
Epoch 54/100
step 391/391 [] - loss: 2.8381 - acc_top1: 0.4361 - acc_top5: 0.7439 - 90ms/step
Eval begin…
step 79/79 [] - loss: 2.6398 - acc_top1: 0.2976 - acc_top5: 0.5876 - 47ms/step
Eval samples: 10000
Epoch 55/100
step 391/391 [] - loss: 1.9737 - acc_top1: 0.4441 - acc_top5: 0.7489 - 91ms/step
Eval begin…
step 79/79 [] - loss: 2.7225 - acc_top1: 0.2969 - acc_top5: 0.5878 - 45ms/step
Eval samples: 10000
Epoch 56/100
step 391/391 [] - loss: 2.1371 - acc_top1: 0.4475 - acc_top5: 0.7523 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.5232 - acc_top1: 0.3012 - acc_top5: 0.5955 - 44ms/step
Eval samples: 10000
Epoch 57/100
step 391/391 [] - loss: 2.0962 - acc_top1: 0.4511 - acc_top5: 0.7565 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.5724 - acc_top1: 0.3050 - acc_top5: 0.5974 - 45ms/step
Eval samples: 10000
Epoch 58/100
step 391/391 [] - loss: 1.9895 - acc_top1: 0.4586 - acc_top5: 0.7648 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.6998 - acc_top1: 0.3015 - acc_top5: 0.5952 - 45ms/step
Eval samples: 10000
Epoch 59/100
step 391/391 [] - loss: 2.0773 - acc_top1: 0.4653 - acc_top5: 0.7685 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.4346 - acc_top1: 0.3036 - acc_top5: 0.5966 - 45ms/step
Eval samples: 10000
Epoch 60/100
step 391/391 [] - loss: 2.3247 - acc_top1: 0.4735 - acc_top5: 0.7756 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.5396 - acc_top1: 0.3053 - acc_top5: 0.5955 - 45ms/step
Eval samples: 10000
Epoch 61/100
step 391/391 [] - loss: 1.6406 - acc_top1: 0.4773 - acc_top5: 0.7760 - 87ms/step
Eval begin…
step 79/79 [] - loss: 3.2420 - acc_top1: 0.2973 - acc_top5: 0.5894 - 45ms/step
Eval samples: 10000
Epoch 62/100
step 391/391 [] - loss: 1.6518 - acc_top1: 0.4863 - acc_top5: 0.7847 - 86ms/step
Eval begin…
step 79/79 [] - loss: 2.6234 - acc_top1: 0.2973 - acc_top5: 0.5957 - 46ms/step
Eval samples: 10000
Epoch 63/100
step 391/391 [] - loss: 2.3919 - acc_top1: 0.4912 - acc_top5: 0.7874 - 90ms/step
Eval begin…
step 79/79 [] - loss: 2.6038 - acc_top1: 0.3006 - acc_top5: 0.5924 - 45ms/step
Eval samples: 10000
Epoch 64/100
step 391/391 [] - loss: 2.2112 - acc_top1: 0.4944 - acc_top5: 0.7904 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.7318 - acc_top1: 0.2889 - acc_top5: 0.5861 - 45ms/step
Eval samples: 10000
Epoch 65/100
step 391/391 [] - loss: 2.1017 - acc_top1: 0.5031 - acc_top5: 0.7968 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.7789 - acc_top1: 0.2967 - acc_top5: 0.5918 - 44ms/step
Eval samples: 10000
Epoch 66/100
step 391/391 [] - loss: 2.0587 - acc_top1: 0.5076 - acc_top5: 0.8031 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.9333 - acc_top1: 0.2832 - acc_top5: 0.5723 - 46ms/step
Eval samples: 10000
Epoch 67/100
step 391/391 [] - loss: 1.9153 - acc_top1: 0.5104 - acc_top5: 0.8038 - 88ms/step
Eval begin…
step 79/79 [] - loss: 3.0879 - acc_top1: 0.3011 - acc_top5: 0.5962 - 44ms/step
Eval samples: 10000
Epoch 68/100
step 391/391 [] - loss: 1.8406 - acc_top1: 0.5135 - acc_top5: 0.8097 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.6704 - acc_top1: 0.3029 - acc_top5: 0.5970 - 46ms/step
Eval samples: 10000
Epoch 69/100
step 391/391 [] - loss: 1.7613 - acc_top1: 0.5165 - acc_top5: 0.8114 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.3267 - acc_top1: 0.3056 - acc_top5: 0.6006 - 44ms/step
Eval samples: 10000
Epoch 70/100
step 391/391 [] - loss: 1.7512 - acc_top1: 0.5238 - acc_top5: 0.8148 - 93ms/step
Eval begin…
step 79/79 [] - loss: 2.1417 - acc_top1: 0.3056 - acc_top5: 0.5988 - 47ms/step
Eval samples: 10000
Epoch 71/100
step 391/391 [] - loss: 1.9838 - acc_top1: 0.5281 - acc_top5: 0.8174 - 86ms/step
Eval begin…
step 79/79 [] - loss: 2.5933 - acc_top1: 0.3185 - acc_top5: 0.6076 - 53ms/step
Eval samples: 10000
Epoch 72/100
step 391/391 [] - loss: 1.6904 - acc_top1: 0.5340 - acc_top5: 0.8231 - 86ms/step
Eval begin…
step 79/79 [] - loss: 2.6118 - acc_top1: 0.3185 - acc_top5: 0.6119 - 45ms/step
Eval samples: 10000
Epoch 73/100
step 391/391 [] - loss: 1.7876 - acc_top1: 0.5394 - acc_top5: 0.8273 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.6305 - acc_top1: 0.3168 - acc_top5: 0.6092 - 45ms/step
Eval samples: 10000
Epoch 74/100
step 391/391 [] - loss: 1.8091 - acc_top1: 0.5470 - acc_top5: 0.8321 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.7842 - acc_top1: 0.3246 - acc_top5: 0.6078 - 46ms/step
Eval samples: 10000
Epoch 75/100
step 391/391 [] - loss: 1.5497 - acc_top1: 0.5534 - acc_top5: 0.8370 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.8507 - acc_top1: 0.3188 - acc_top5: 0.6101 - 45ms/step
Eval samples: 10000
Epoch 76/100
step 391/391 [] - loss: 2.1434 - acc_top1: 0.5572 - acc_top5: 0.8404 - 86ms/step
Eval begin…
step 79/79 [] - loss: 2.8459 - acc_top1: 0.3149 - acc_top5: 0.6078 - 46ms/step
Eval samples: 10000
Epoch 77/100
step 391/391 [] - loss: 1.5916 - acc_top1: 0.5660 - acc_top5: 0.8453 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.2425 - acc_top1: 0.3205 - acc_top5: 0.6156 - 47ms/step
Eval samples: 10000
Epoch 78/100
step 391/391 [] - loss: 1.9925 - acc_top1: 0.5728 - acc_top5: 0.8486 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.2278 - acc_top1: 0.3160 - acc_top5: 0.6146 - 45ms/step
Eval samples: 10000
Epoch 79/100
step 391/391 [] - loss: 1.7550 - acc_top1: 0.5779 - acc_top5: 0.8541 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.3391 - acc_top1: 0.3269 - acc_top5: 0.6111 - 47ms/step
Eval samples: 10000
Epoch 80/100
step 391/391 [] - loss: 1.7625 - acc_top1: 0.5830 - acc_top5: 0.8574 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.5607 - acc_top1: 0.3062 - acc_top5: 0.6047 - 45ms/step
Eval samples: 10000
Epoch 81/100
step 391/391 [] - loss: 1.2804 - acc_top1: 0.5859 - acc_top5: 0.8603 - 86ms/step
Eval begin…
step 79/79 [] - loss: 2.6926 - acc_top1: 0.3219 - acc_top5: 0.6136 - 46ms/step
Eval samples: 10000
Epoch 82/100
step 391/391 [] - loss: 1.5730 - acc_top1: 0.5913 - acc_top5: 0.8651 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.4792 - acc_top1: 0.3106 - acc_top5: 0.6033 - 45ms/step
Eval samples: 10000
Epoch 83/100
step 391/391 [] - loss: 1.7747 - acc_top1: 0.5996 - acc_top5: 0.8669 - 86ms/step
Eval begin…
step 79/79 [] - loss: 2.4888 - acc_top1: 0.3114 - acc_top5: 0.5989 - 44ms/step
Eval samples: 10000
Epoch 84/100
step 391/391 [] - loss: 1.3741 - acc_top1: 0.6046 - acc_top5: 0.8708 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.5207 - acc_top1: 0.3173 - acc_top5: 0.6019 - 45ms/step
Eval samples: 10000
Epoch 85/100
step 391/391 [] - loss: 1.6394 - acc_top1: 0.6142 - acc_top5: 0.8751 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.9659 - acc_top1: 0.3062 - acc_top5: 0.5897 - 45ms/step
Eval samples: 10000
Epoch 86/100
step 391/391 [] - loss: 1.8119 - acc_top1: 0.6169 - acc_top5: 0.8782 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.5617 - acc_top1: 0.3052 - acc_top5: 0.6028 - 45ms/step
Eval samples: 10000
Epoch 87/100
step 391/391 [] - loss: 1.1349 - acc_top1: 0.6194 - acc_top5: 0.8820 - 86ms/step
Eval begin…
step 79/79 [] - loss: 2.7186 - acc_top1: 0.3088 - acc_top5: 0.5979 - 49ms/step
Eval samples: 10000
Epoch 88/100
step 391/391 [] - loss: 1.4958 - acc_top1: 0.6297 - acc_top5: 0.8854 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.3525 - acc_top1: 0.3133 - acc_top5: 0.6022 - 45ms/step
Eval samples: 10000
Epoch 89/100
step 391/391 [] - loss: 1.5459 - acc_top1: 0.6318 - acc_top5: 0.8883 - 86ms/step
Eval begin…
step 79/79 [] - loss: 2.6850 - acc_top1: 0.3091 - acc_top5: 0.6042 - 46ms/step
Eval samples: 10000
Epoch 90/100
step 391/391 [] - loss: 1.4070 - acc_top1: 0.6391 - acc_top5: 0.8911 - 90ms/step
Eval begin…
step 79/79 [] - loss: 1.8997 - acc_top1: 0.3032 - acc_top5: 0.5996 - 45ms/step
Eval samples: 10000
Epoch 91/100
step 391/391 [] - loss: 1.7658 - acc_top1: 0.6371 - acc_top5: 0.8925 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.6297 - acc_top1: 0.3061 - acc_top5: 0.5952 - 46ms/step
Eval samples: 10000
Epoch 92/100
step 391/391 [] - loss: 1.1509 - acc_top1: 0.6494 - acc_top5: 0.8982 - 91ms/step
Eval begin…
step 79/79 [] - loss: 2.4249 - acc_top1: 0.3208 - acc_top5: 0.6074 - 46ms/step
Eval samples: 10000
Epoch 93/100
step 391/391 [] - loss: 1.1956 - acc_top1: 0.6519 - acc_top5: 0.8989 - 89ms/step
Eval begin…
step 79/79 [] - loss: 2.7847 - acc_top1: 0.3181 - acc_top5: 0.6033 - 48ms/step
Eval samples: 10000
Epoch 94/100
step 391/391 [] - loss: 1.6424 - acc_top1: 0.6524 - acc_top5: 0.9005 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.5296 - acc_top1: 0.3200 - acc_top5: 0.6046 - 46ms/step
Eval samples: 10000
Epoch 95/100
step 391/391 [] - loss: 1.2563 - acc_top1: 0.6582 - acc_top5: 0.9028 - 85ms/step
Eval begin…
step 79/79 [] - loss: 2.5143 - acc_top1: 0.3215 - acc_top5: 0.6111 - 47ms/step
Eval samples: 10000
Epoch 96/100
step 391/391 [] - loss: 1.4293 - acc_top1: 0.6614 - acc_top5: 0.9047 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.1475 - acc_top1: 0.3281 - acc_top5: 0.6183 - 45ms/step
Eval samples: 10000
Epoch 97/100
step 391/391 [] - loss: 1.2284 - acc_top1: 0.6714 - acc_top5: 0.9101 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.7041 - acc_top1: 0.3215 - acc_top5: 0.6118 - 46ms/step
Eval samples: 10000
Epoch 98/100
step 391/391 [] - loss: 1.2298 - acc_top1: 0.6817 - acc_top5: 0.9140 - 87ms/step
Eval begin…
step 79/79 [] - loss: 2.5701 - acc_top1: 0.3266 - acc_top5: 0.6145 - 47ms/step
Eval samples: 10000
Epoch 99/100
step 391/391 [] - loss: 1.2673 - acc_top1: 0.6866 - acc_top5: 0.9168 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.4477 - acc_top1: 0.3184 - acc_top5: 0.6114 - 45ms/step
Eval samples: 10000
Epoch 100/100
step 391/391 [] - loss: 1.2454 - acc_top1: 0.6918 - acc_top5: 0.9195 - 88ms/step
Eval begin…
step 79/79 [] - loss: 2.8776 - acc_top1: 0.3278 - acc_top5: 0.6141 - 45ms/step
Eval samples: 10000
五.实验对比
可以看出，加入注意力机制比不加入注意力机制的效果更好，精度更高，适当增大epoch,效果应当更显著

5.1 结果可视化
图一：resnet50 Top1、5准确率在这里插入图片描述
图二：resnet50-NAM Top1、5准确率
5.2 消融实验
本项目对比了ResNet50和ResNet-NAM的对比可以看出，ResNet50-NAM效果比ResNet50好；然而epoch只设置为100，后续增大epoch效果应该会更加显著
六、总结
1.感谢paddlepaddle组织的本次AI达人特训营活动

2.本次论文NAM注意力机制可谓即插即用，我们是在BottleNeck的Block块中，最后一层卷积后插入NAM模块，经过100个epoch在cifar100数据上效果还不错！

3.我们建议增大epoch至200，或者300，模型将有更好的精度效果

附录
论文：https://arxiv.org/pdf/2111.12419.pdf

NAM注意力机制paddle版本：

https://aistudio.baidu.com/aistudio/projectdetail/4190589?contributionType=1

Github项目地址：https://github.com/Christian-lyc/NAM/blob/main/MODELS/model_resnet.py#L88

In [ ]
请点击此处查看本环境基本用法.
Please click here for more detailed instructions.

百度飞桨AI Studio社区

学大模型，用大模型上飞桨星河社区！每天8点V100G算力免费领！免费领取ERNIE 4.0 100w Token >>>

更多推荐

利用Amazon Bedrock畅玩Claude 3等多种领先模型，抢占AI高地(体验倒计时4小时)

百度飞桨星河社区

RAPTOR：索引树状 RAG，使用树结构来捕捉文本的高级和低级细节

百度飞桨星河社区

MultiHop-RAG：多跳查询的基准检索增强生成

百度飞桨星河社区

所有评论(0)

查看更多评论

AI Studio

@m0_63642362

已为社区贡献1438条内容

【AI达人特训营】基于ResNet50的NAM注意力机制论文复现

AI Studio

Layer (type) Input Shape Output Shape Param #

Total params: 25,610,152 Trainable params: 25,503,912 Non-trainable params: 106,240

Input size (MB): 0.01 Forward/backward pass size (MB): 5.36 Params size (MB): 97.69 Estimated Total Size (MB): 103.07

model.prepare(paddle.optimizer.SGD(learning_rate=0.001, parameters=model.parameters()),#使用Adam优化器，学习率为0.0001

paddle.nn.CrossEntropyLoss(),#损失函数使用交叉熵函数

paddle.metric.Accuracy())#Acc用top1与top5精准度表示

#模型可视化 model.summary((-1, 3, 32, 32))

Layer (type) Input Shape Output Shape Param #

Total params: 23,818,788 Trainable params: 23,652,132 Non-trainable params: 166,656

Input size (MB): 0.01 Forward/backward pass size (MB): 121.64 Params size (MB): 90.86 Estimated Total Size (MB): 212.51

所有评论(0)

AI Studio

Total params: 25,610,152
Trainable params: 25,503,912
Non-trainable params: 106,240

Input size (MB): 0.01
Forward/backward pass size (MB): 5.36
Params size (MB): 97.69
Estimated Total Size (MB): 103.07

#模型可视化
model.summary((-1, 3, 32, 32))

Total params: 23,818,788
Trainable params: 23,652,132
Non-trainable params: 166,656

Input size (MB): 0.01
Forward/backward pass size (MB): 121.64
Params size (MB): 90.86
Estimated Total Size (MB): 212.51