基于飞桨复现RAM

Mnist28*28, RAM(Recurrent Models of Visual Attention), 6 glimpses, 8 × 8, 1 scale, test_err 1.17%

AI Studio

280人浏览 · 2022-02-13 14:23:02

AI Studio · 2022-02-13 14:23:02 发布

一、论文简介

1.1 简介

Recurrent Attention Model (RAM)，它能顺序处理输入信息，在每个时间步关注图像内不同的子区域，然后增量式的结合来自这些固定位置的信息，并建立图像的动态内部表示。

RAM的优点在于能自主选择图像的子区域进行处理，而不像传统的卷积模型一样复杂度随着输入图像像素变大而线性增长。

论文地址

1.2 网络结构

本文将注意力问题建模为目标导向的agent与视觉环境交互的序列决策过程，agent的核心是一个循环神经网络，它在每一时间步处理来自sensor收集的子图信息，并随着时间推移集成图像信息，并选择如何行动和部署下一个时间步的sensor。

RAM模型结构如上图所示，其中包含如下五个部分：

glimpse sensor:glimpse sensor受到视网膜注意力机制的启发，即人往往能清晰的看见所关注对象的细节（内容少，高分辨率），同时保留对背景的模糊感受（内容多，低分辨率）。于是设计的glimpse sensor能从图像 x中提取漏斗状的一瞥（glimpse）phi，sensor首先编码靠近位置l的一块高像素的小区域，然后再渐进的从l附近取更大且像素更低的子区域（所有的截取的子区域需要缩放到同一大小，所以大图像素低），从而得到原始图像 x的压缩表示；
- 下面第一张图是截取位置l附近不同尺度的区域，然后第二章是将他们缩放到同一尺度，使得细节部分有高分辨率，背景低分辨率。
glimpse network: 该网络将sensor得到的压缩表示"what" (phi)和位置信息"where" (l)结合起来，得到这一瞥的特征向量g_t；
core network: 核心网络是个循环神经网络，该网络维持一个内部状态 h_t ，代表从过去观测历史中提取的整合信息。它通过状态向量 h_t 编码angent对环境的知识，并且在每个时间步 t都会更新。时间步t时的输入为上一个时刻glimpse向量g_(t-1)和状态向量h_(t-1)；
location network:位置网络，使用rnn状态向量h_t，在时间步t时产生shape为[bsz,2]的位置坐标l_t，再同输入图像 x送入glimpse得到输入向量g_(t+1),同状态向量h_t作为t+1时刻rnn的输入；
action network: 在固定数的时间步之后，使用rnn的内部状态‘h_t’生成最终的分类输出 y。

总的来说，RAM是围绕rnn展开的，输入是glimpse向量和t时刻状态向量，输出是t+1时刻状态向量，代表集成的图像信息。利用状态向量输入两个子网络location和action 可以得到两个输出：l_t和a_t，l_t用于指导sensor截取子图并编码为输入向量，a_t用来完成分类任务。

二、复现结果

2.1 实验结果

本项目使用28x28的MNIST数据集来复现，RAM模型包含6个glimpses，patch_size为8x8，缩放因子scale为1，论文中指标为：

本项目的验证误差为1.18%（290epoch），原文和本项目在MNIST测试集上的误差为：

Task	Paper	Me
28x28 MNIST	1.29%	1.17%~1.28%

本项目的模型权重ram_6_8x8_1_model_best.pdparams（aistudio上zip里面有）已经上传到百度网盘:链接 ,提取码：v6d3

2.2 实验环境以及超参

NO.	Paddle Version	Memory	Card	Batch Size	Learning Rate	LR Factor	LR Patience	Epoch	Training time	val err	test err
01	2.1.2	16G	V100*1	128	3e-4	0.8	20	290	~2h	1.15%	1.17%
02	2.1.2	16G	V100*1	128	3e-4	0.8	20	315	~3h	1.033 %	1.28%

注：

第一次是先用factor=0.1,patience=20训练了200轮，发现142轮达到最优，未达到指定精度，且后面学习率过小为0了。于是从142轮开始恢复训练，初始学习率仍为3e-4，然后factor=0.8，patience=10,
继续训练到290轮，详细见logs里RAM_local290.log日志。且该指标是在本地3060环境达到精度，3060一轮约45s，v100一轮约30s。

第二次是在aistudio上，初始学习率3e-4，然后factor=0.8，patience=10,训练到200轮，发现第192轮best，test acc为1.68，然后恢复训练，到315轮时验证误差最小为1.033%，于是停止训练，评估得到1.28%

三、准备工作

# 解压代码
!unzip RAM.zip

Archive:  RAM.zip
replace RAM/align.py? [y]es, [n]o, [A]ll, [N]one, [r]ename: ^C

# 进入目录，并安装依赖
%cd RAM/
!pip install tensorboard_logger

/home/aistudio/RAM
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: tensorboard_logger in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (0.1.0)
Requirement already satisfied: six in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from tensorboard_logger) (1.15.0)
Requirement already satisfied: scipy>=0.19.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from tensorboard_logger) (1.6.3)
Requirement already satisfied: protobuf in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from tensorboard_logger) (3.14.0)
Requirement already satisfied: numpy in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from tensorboard_logger) (1.20.3)
Requirement already satisfied: pillow>=4.1.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from tensorboard_logger) (7.1.2)

目录结构

.
├── README.md
├── align.py # 转换权重
├── ckpt # 权重
│   └── ram_6_8x8_1_model_best.pdparams
├── config.py # 配置文件
├── data # 数据
│   ├── MNIST
├── data_loader.py # 加载数据
├── logs # 日志
├── main.py # 主函数
├── model.py # RAM主体模型
├── modules.py # RAM5个部分
├── plot_glimpses.py # 画图
├── plots # 图片
├── requirements.txt
├── trainer.py # 训练、评估函数
└── utils.py # 工具

四、模型训练

!python main.py

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sized
W1123 12:01:46.425436   726 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.0, Runtime API Version: 10.1
W1123 12:01:46.429379   726 device_context.cc:465] device: 0, cuDNN Version: 7.6.
[*] Model Checkpoint Dir: ./ckpt
[*] Param Path: ./ckpt/ram_6_8x8_1_params.json
2021-11-23 12:01:49,497 | RAM: 
[*] Train on 54000 samples, validate on 6000 samples
INFO:RAM:
[*] Train on 54000 samples, validate on 6000 samples
2021-11-23 12:01:49,497 | RAM: 
Epoch: 1/200 - LR: 0.000300
INFO:RAM:
Epoch: 1/200 - LR: 0.000300
  0%|                                                 | 0/54000 [00:00<?, ?it/s]/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/tensor/creation.py:130: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
8.6s - loss: 2.044 - acc: 36.719:  24%|▏| 12800/54000 [00:08<00:26, 1552.31it/s]2021-11-23 12:01:58,055 | RAM: 8.6s - loss: 2.044 - acc: 36.719
INFO:RAM:8.6s - loss: 2.044 - acc: 36.719
11.5s - loss: 2.239 - acc: 21.875:  33%|▎| 17920/54000 [00:11<00:19, 1840.30it/s]

五、模型评估

## aistudio上训练的，315轮，验证误差1.033，测试误差1.28%
!python main.py --is_train=False --best True --ckpt_dir ckpt_aistudio

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sized
W1124 09:46:43.853452   793 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W1124 09:46:43.857133   793 device_context.cc:465] device: 0, cuDNN Version: 7.6.
2021-11-24 09:46:48,040 | RAM: [*] Loading model from ckpt_aistudio
INFO:RAM:[*] Loading model from ckpt_aistudio
2021-11-24 09:46:48,050 | RAM: [*] Loaded ram_6_8x8_1_model_best.pdparams checkpoint @ epoch 315 with best valid acc of 98.917
INFO:RAM:[*] Loaded ram_6_8x8_1_model_best.pdparams checkpoint @ epoch 315 with best valid acc of 98.917
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/tensor/creation.py:130: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
2021-11-24 09:46:53,835 | RAM: [*] Test Acc: 9872.0/10000 (98.72% - 1.28%)
INFO:RAM:[*] Test Acc: 9872.0/10000 (98.72% - 1.28%)

## 本地3060训练的，见日志logs/RAM_local290.log，290轮，验证误差1.15，测试误差1.17%
7%
!python main.py --is_train=False --best True

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sized
INFO:matplotlib.font_manager:font search path ['/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/ttf', '/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/afm', '/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/mpl-data/fonts/pdfcorefonts']
INFO:matplotlib.font_manager:generated new fontManager
Cache file /home/aistudio/.cache/paddle/dataset/mnist/t10k-images-idx3-ubyte.gz not found, downloading https://dataset.bj.bcebos.com/mnist/t10k-images-idx3-ubyte.gz 
Begin to download
item 403/403 [============================>.] - ETA: 0s - 389us/it
Download finished
Cache file /home/aistudio/.cache/paddle/dataset/mnist/t10k-labels-idx1-ubyte.gz not found, downloading https://dataset.bj.bcebos.com/mnist/t10k-labels-idx1-ubyte.gz 
Begin to download
item 2/2 [===========================>..] - ETA: 0s - 642us/it
Download finished
W1124 09:37:17.144690   311 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W1124 09:37:17.149226   311 device_context.cc:465] device: 0, cuDNN Version: 7.6.
2021-11-24 09:37:22,326 | RAM: [*] Loading model from ./ckpt
INFO:RAM:[*] Loading model from ./ckpt
2021-11-24 09:37:22,336 | RAM: [*] Loaded ram_6_8x8_1_model_best.pdparams checkpoint @ epoch 290 with best valid acc of 98.917
INFO:RAM:[*] Loaded ram_6_8x8_1_model_best.pdparams checkpoint @ epoch 290 with best valid acc of 98.917
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/tensor/creation.py:130: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
2021-11-24 09:37:28,007 | RAM: [*] Test Acc: 9883.0/10000 (98.83% - 1.17%)
INFO:RAM:[*] Test Acc: 9883.0/10000 (98.83% - 1.17%)

六、总结

这里就说下复现遇到的小坑：

1.rsample（location网络）：
paddle.distribution.Normal，没有torch.distribution.Normal().rsample方法，
参考torch源码实现后，在对齐精度时发现不能完全对齐，该操作有随机性，差0.3%左右，不过影响不大；

    def rsample(loc, scale):
      shape = loc.shape
      normal_ = paddle.nn.initializer.Normal()
      eps = paddle.empty(shape, dtype=loc.dtype)
      normal_(eps)
      return loc + eps * scale

2.索引：

在glimpse的retina的extract_patch方法内，根据输入的位置信息lt[bsz,2]，对图片进行采样（8*8patch）。

    def extract_patch(self, x, l, size):
			...
        patch = []
        for i in range(B):
            subset=x[i, :, start[i, 1] : end[i, 1], start[i, 0] : end[i, 0]]
            patch.append(subset)
        return paddle.to_tensor(np.stack(patch))