「复现赛」A TRANSFORMER-BASED SIAMESE NETWORK
「复现赛」A TRANSFORMER-BASED SIAMESE NETWORK
·
「论文复现」A TRANSFORMER-BASED SIAMESE NETWORK FOR CHANGE DETECTION
Index Terms：
Change detection, transformer - Siamese network, attention mechanism, multilayer perceptron, remote
sensing.
- 论文地址：A TRANSFORMER-BASED SIAMESE NETWORK FOR CHANGE DETECTION
- 参考github：https://github.com/wgcban/ChangeFormer
- 复现地址：https://github.com/HULEIYI/ChangeFormer-pd
(一) 摘要
1.1 方法简介
ChangeFormer是一种基于Transformer的Siamese网络架构，用于从一对前后时序的遥感图像中提取变化区域（ChangeDetection,CD）。与基于全卷积网络（ConvNet）的CD框架不同，ChangeFormer将层次化的Transformer编码器与多层感知（MLP）解码器在Siamese网络统一起来，以高效提取准确CD所需的多尺度和长程依赖的细节。对两个CD数据集的实验（Paddle版本仅在LEVIR-CD数据集上进行了训练和推理测试）表明，与之前的版本相比，端到端可训练的ChangeFormer架构实现了更好的CD性能。原论文代码和预训练模型可在以上参考链接找到。
1.2 模型概览
注：以下ChangeFormer的有关分析参考CSDN博客：【论文笔记】A Transformer-based Siamese network for change detection
-
ChangeFormer的主要贡献是将Transformer模块全程应用于变化检测（CD），而不是使用CNN+TRM的网络搭建方式，证明了纯TRM网络+MLP Head也能取得SOTA效果。网络整体架构可以分为三个模块：
- hierarchical transformer encoder：构成孪生TRM网络，用于特征提取和编码；
- difference modules：四个差异化模块，计算多尺度差异；
- light weight MLP decoder: 轻量化的MLP decoder，融合多尺度特征，进行CD mask的预测。
-
网络的pipeline：
- 输入变化检测图像对，孪生TRM网络分别提取前后时相的多级特征；
- difference modules接收孪生网络不同层生成的共4组不同尺度的特征，进行拼接和融合；
- 将上一步产生的多级特征输入MLP 解码器进行聚合，得到变化检测mask输出。
（二）模型架构
从2020年ViT模型横空出世到如今Vision Transformer大行其道，且有一统CV的态势，也只有短短两年时间。现今各大CV任务榜单上均不乏ViT，SwinTRM以及其他一些TRM变体的身影。
Transformer能够在NLP和CV领域大放异彩，主要来源于TRM的全局注意力以及神经元间长程依赖的建模能力。有关TRM的介绍特别是视觉TRM的有关知识，可以参见AI Studio课程：从零开始学视觉Transformer。
即使TRM在各自然图像任务上应用广泛且效果优异，但是在遥感类任务上却有所滞后，以此篇论文关注的变化检测领域来说，BiT引入TRM完成CD任务，但backbone仍是CNN（ResNet18）（有关BiT的介绍以及上手体验可以转【第六期论文复现赛-变化检测】Remote Sensing Image Change Detection with Transformers），直到本次复现论文ChangeFormer才完全使用TRM架构作为模型主体。下面就让笔者带着大家一探ChangeFormer里TRM的魅力吧。
模型的主体架构在章节（一）中已经说明，不再赘述，下文将就这三种模块进行说明以及代码展示。
2.1 hierarchical transformer encoder
-
hierachical transformer encoder是ChangeFormer Encoder部分的基础模块，用来构建孪生特征提取网络。
- 孪生网络简单来说就是共享参数的网络处理一对图像对来完成某种任务的backbone框架。关于孪生网络的介绍可以参见[【第六期论文复现赛-变化检测】SNUNet-CD.
-
ChangeFormer的hierachical transformer encoder里的TRM模块大体上就是经典的ViT模块，最主要的修改即其在堆叠的TRM层间加入了Downsample模块。
- DownSample模块的主要功能就是对patch进行下采样，模仿CNN结构获得多尺度的图像特征。
- DownSample模块的主要功能就是对patch进行下采样，模仿CNN结构获得多尺度的图像特征。
-
Encoder代码如下：
# Transormer Ecoder with x2, x4, x8, x16 scales
class EncoderTransformer_v3(nn.Layer):
def __init__(
self,
img_size=256,
patch_size=3,
in_chans=3,
num_classes=2,
embed_dims=[32,64,128,256],
num_heads=[2,2,4,8],
mlp_ratios=[4,4,4,4],
qkv_bias=True,
qk_scale=None,
drop_rate=0.,
attn_drop_rate=0.,
drop_path_rate=0.,
norm_layer=nn.LayerNorm,
depths=[3,3,6,18],
sr_ratios=[8,4,2,1]):
super().__init__()
self.num_classes = num_classes
self.depths = depths
self.embed_dims = embed_dims
# patch embedding definitions
self.patch_embed1 = OverlapPatchEmbed(
img_size=img_size,
patch_size=7,
stride=4,
in_chans=in_chans,
embed_dim=embed_dims[0])
self.patch_embed2 = OverlapPatchEmbed(
img_size=img_size // 4,
patch_size=patch_size,
stride=2,
in_chans=embed_dims[0],
embed_dim=embed_dims[1])
self.patch_embed3 = OverlapPatchEmbed(
img_size=img_size // 8,
patch_size=patch_size,
stride=2,
in_chans=embed_dims[1],
embed_dim=embed_dims[2])
self.patch_embed4 = OverlapPatchEmbed(
img_size=img_size // 16,
patch_size=patch_size,
stride=2,
in_chans=embed_dims[2],
embed_dim=embed_dims[3])
# Stage-1 (x1/4 scale)
dpr = [x.item() for x in pd.linspace(0, drop_path_rate, sum(depths))]
cur = 0
self.block1 = nn.LayerList([Block(dim=embed_dims[0],
num_heads=num_heads[0],
mlp_ratio=mlp_ratios[0],
qkv_bias=qkv_bias,
qk_scale=qk_scale,
drop=drop_rate,
attn_drop=attn_drop_rate,
drop_path=dpr[cur + i],
norm_layer=norm_layer,
sr_ratio=sr_ratios[0]) for i in range(depths[0])])
self.norm1 = norm_layer(embed_dims[0])
# Stage-2 (x1/8 scale)
cur += depths[0]
self.block2 = nn.LayerList([Block(dim=embed_dims[1],
num_heads=num_heads[1],
mlp_ratio=mlp_ratios[1],
qkv_bias=qkv_bias,
qk_scale=qk_scale,
drop=drop_rate,
attn_drop=attn_drop_rate,
drop_path=dpr[cur + i],
norm_layer=norm_layer,
sr_ratio=sr_ratios[1]) for i in range(depths[1])])
self.norm2 = norm_layer(embed_dims[1])
# Stage-3 (x1/16 scale)
cur += depths[1]
self.block3 = nn.LayerList([Block(dim=embed_dims[2],
num_heads=num_heads[2],
mlp_ratio=mlp_ratios[2],
qkv_bias=qkv_bias,
qk_scale=qk_scale,
drop=drop_rate,
attn_drop=attn_drop_rate,
drop_path=dpr[cur + i],
norm_layer=norm_layer,
sr_ratio=sr_ratios[2]) for i in range(depths[2])])
self.norm3 = norm_layer(embed_dims[2])
# Stage-4 (x1/32 scale)
cur += depths[2]
self.block4 = nn.LayerList([Block(dim=embed_dims[3],
num_heads=num_heads[3],
mlp_ratio=mlp_ratios[3],
qkv_bias=qkv_bias,
qk_scale=qk_scale,
drop=drop_rate,
attn_drop=attn_drop_rate,
drop_path=dpr[cur + i],
norm_layer=norm_layer,
sr_ratio=sr_ratios[3]) for i in range(depths[3])])
self.norm4 = norm_layer(embed_dims[3])
self.apply(self._init_weights)
# for ent in self.parameters():
# self._init_weights(ent)
def _init_weights(self, m):
if isinstance(m, nn.Linear):
trunc_normal_op = nn.initializer.TruncatedNormal(std=.02)
trunc_normal_op(m.weight)
# trunc_normal_(m.weight, std=.02)
if isinstance(m, nn.Linear) and m.bias is not None:
init_bias = nn.initializer.Constant(0)
init_bias(m.bias)
# nn.init.constant_(m.bias, 0)
elif isinstance(m, nn.LayerNorm):
init_bias = nn.initializer.Constant(0)
init_bias(m.bias)
# nn.init.constant_(m.bias, 0)
init_weight = nn.initializer.Constant(1.0)
init_weight(m.weight)
# nn.init.constant_(m.weight, 1.0)
elif isinstance(m, nn.Conv2D):
fan_out = m._kernel_size[0] * m._kernel_size[1] * m._out_channels
fan_out //= m._groups
init_weight = nn.initializer.Normal(0, math.sqrt(2.0 / fan_out))
init_weight(m.weight)
# m.weight.data.normal_(0, math.sqrt(2.0 / fan_out))
if m.bias is not None:
init_bias = nn.initializer.Constant(0)
init_bias(m.bias)
# m.bias.data.zero_()
def reset_drop_path(self, drop_path_rate):
dpr = [
x.item() for x in pd.linspace(
0, drop_path_rate, sum(
self.depths))]
cur = 0
for i in range(self.depths[0]):
self.block1[i].drop_path.drop_prob = dpr[cur + i]
cur += self.depths[0]
for i in range(self.depths[1]):
self.block2[i].drop_path.drop_prob = dpr[cur + i]
cur += self.depths[1]
for i in range(self.depths[2]):
self.block3[i].drop_path.drop_prob = dpr[cur + i]
cur += self.depths[2]
for i in range(self.depths[3]):
self.block4[i].drop_path.drop_prob = dpr[cur + i]
def forward_features(self, x):
# print(x)
B = x.shape[0]
outs = []
# stage 1
x1, H1, W1 = self.patch_embed1(x)
# print(x1,H1,W1)
for i, blk in enumerate(self.block1):
x1 = blk(x1, H1, W1)
# print(x1)
x1 = self.norm1(x1)
x1 = x1.reshape([B, H1, W1, -1]).transpose([0, 3, 1, 2])
# print(x1)
outs.append(x1)
# stage 2
x1, H1, W1 = self.patch_embed2(x1)
for i, blk in enumerate(self.block2):
x1 = blk(x1, H1, W1)
x1 = self.norm2(x1)
x1 = x1.reshape([B, H1, W1, -1]).transpose([0, 3, 1, 2])
outs.append(x1)
# stage 3
x1, H1, W1 = self.patch_embed3(x1)
for i, blk in enumerate(self.block3):
x1 = blk(x1, H1, W1)
x1 = self.norm3(x1)
x1 = x1.reshape([B, H1, W1, -1]).transpose([0, 3, 1, 2])
outs.append(x1)
# stage 4
x1, H1, W1 = self.patch_embed4(x1)
for i, blk in enumerate(self.block4):
x1 = blk(x1, H1, W1)
x1 = self.norm4(x1)
x1 = x1.reshape([B, H1, W1, -1]).transpose([0, 3, 1, 2])
outs.append(x1)
return outs
def forward(self, x):
x = self.forward_features(x)
return x
2.2 difference module
-
difference module的主要功能就是在不同尺度进行特征拼接输出差异。
- 基本过程是进行拼接，然后卷积。
- 要注意的是，这一操作并没有直接计算特征图的绝对差值，而是在训练过中学习每个尺度上的最优距离度量。
-
differenc module代码：
# Difference Layer
def conv_diff(in_channels, out_channels):
return nn.Sequential(
nn.Conv2D(in_channels, out_channels, kernel_size=3, padding=1),
nn.ReLU(),
nn.BatchNorm2D(out_channels),
nn.Conv2D(out_channels, out_channels, kernel_size=3, padding=1),
nn.ReLU()
)
2.3 MLP decoder
-
MLP Decoder的操作可以分为如下三个步骤：
- MLP & Upsampling
- MLP层统一不同特征图的通道数，然后，上采样到H/4,W/4。
- Concatenation & Fusion
- 将上一步得到的上采样特征图进行拼接，然后再使用一层MLP层进行融合。
- Upsampling & Classification
- 再次进行上采样，得到H,W尺寸的特征图，最后再使用一层MLP层得到CD mask。
- MLP & Upsampling
-
MLP decoder的代码：
class DecoderTransformer_v3(nn.Layer):
"""
Transformer Decoder
"""
def __init__(
self,
input_transform='multiple_select',
in_index=[0,1,2,3],
align_corners=True,
in_channels=[32,64,128,256],
embedding_dim=64,
output_nc=2,
decoder_softmax=False,
feature_strides=[2,4,8,16]):
super(DecoderTransformer_v3, self).__init__()
# assert
assert len(feature_strides) == len(in_channels)
assert min(feature_strides) == feature_strides[0]
# settings
self.feature_strides = feature_strides
self.input_transform = input_transform
self.in_index = in_index
self.align_corners = align_corners
self.in_channels = in_channels
self.embedding_dim = embedding_dim
self.output_nc = output_nc
c1_in_channels, c2_in_channels, c3_in_channels, c4_in_channels = self.in_channels
# MLP decoder heads
self.linear_c4 = MLP(
input_dim=c4_in_channels,
embed_dim=self.embedding_dim)
self.linear_c3 = MLP(
input_dim=c3_in_channels,
embed_dim=self.embedding_dim)
self.linear_c2 = MLP(
input_dim=c2_in_channels,
embed_dim=self.embedding_dim)
self.linear_c1 = MLP(
input_dim=c1_in_channels,
embed_dim=self.embedding_dim)
# convolutional Difference Layers
self.diff_c4 = conv_diff(
in_channels=2 * self.embedding_dim,
out_channels=self.embedding_dim)
self.diff_c3 = conv_diff(
in_channels=2 * self.embedding_dim,
out_channels=self.embedding_dim)
self.diff_c2 = conv_diff(
in_channels=2 * self.embedding_dim,
out_channels=self.embedding_dim)
self.diff_c1 = conv_diff(
in_channels=2 * self.embedding_dim,
out_channels=self.embedding_dim)
# taking outputs from middle of the encoder
self.make_pred_c4 = make_prediction(
in_channels=self.embedding_dim,
out_channels=self.output_nc)
self.make_pred_c3 = make_prediction(
in_channels=self.embedding_dim,
out_channels=self.output_nc)
self.make_pred_c2 = make_prediction(
in_channels=self.embedding_dim,
out_channels=self.output_nc)
self.make_pred_c1 = make_prediction(
in_channels=self.embedding_dim,
out_channels=self.output_nc)
# Final linear fusion layer
self.linear_fuse = nn.Sequential(
nn.Conv2D(
in_channels=self.embedding_dim *
len(in_channels),
out_channels=self.embedding_dim,
kernel_size=1),
nn.BatchNorm2D(
self.embedding_dim))
# Final predction head
self.convd2x = UpsampleConvLayer(
self.embedding_dim,
self.embedding_dim,
kernel_size=4,
stride=2)
self.dense_2x = nn.Sequential(ResidualBlock(self.embedding_dim))
self.convd1x = UpsampleConvLayer(
self.embedding_dim,
self.embedding_dim,
kernel_size=4,
stride=2)
self.dense_1x = nn.Sequential(ResidualBlock(self.embedding_dim))
self.change_probability = ConvLayer(
self.embedding_dim,
self.output_nc,
kernel_size=3,
stride=1,
padding=1)
# Final activation
self.output_softmax = decoder_softmax
self.active = nn.Sigmoid()
def _transform_inputs(self, inputs):
"""Transform inputs for decoder.
Args:
inputs (list[Tensor]): List of multi-level img features.
Returns:
Tensor: The transformed inputs
"""
if self.input_transform == 'resize_concat':
inputs = [inputs[i] for i in self.in_index]
upsampled_inputs = [
resize(
input=x,
size=inputs[0].shape[2:],
mode='bilinear',
align_corners=self.align_corners) for x in inputs
]
inputs = pd.concat(upsampled_inputs, dim=1)
elif self.input_transform == 'multiple_select':
inputs = [inputs[i] for i in self.in_index]
else:
inputs = inputs[self.in_index]
return inputs
def forward(self, inputs1, inputs2):
# Transforming encoder features (select layers)
x_1 = self._transform_inputs(inputs1) # len=4, 1/2, 1/4, 1/8, 1/16
x_2 = self._transform_inputs(inputs2) # len=4, 1/2, 1/4, 1/8, 1/16
# img1 and img2 features
c1_1, c2_1, c3_1, c4_1 = x_1
c1_2, c2_2, c3_2, c4_2 = x_2
############## MLP decoder on C1-C4 ###########
n, _, h, w = c4_1.shape
outputs = []
# Stage 4: x1/32 scale
_c4_1 = self.linear_c4(c4_1).transpose([0, 2, 1]).reshape(
[n, -1, c4_1.shape[2], c4_1.shape[3]])
_c4_2 = self.linear_c4(c4_2).transpose([0, 2, 1]).reshape(
[n, -1, c4_2.shape[2], c4_2.shape[3]])
_c4 = self.diff_c4(pd.concat((_c4_1, _c4_2), axis=1))
p_c4 = self.make_pred_c4(_c4)
outputs.append(p_c4)
_c4_up = resize(_c4,
size=c1_2.shape[2:],
mode='bilinear',
align_corners=False)
# Stage 3: x1/16 scale
_c3_1 = self.linear_c3(c3_1).transpose([0, 2, 1]).reshape(
[n, -1, c3_1.shape[2], c3_1.shape[3]])
_c3_2 = self.linear_c3(c3_2).transpose([0, 2, 1]).reshape(
[n, -1, c3_2.shape[2], c3_2.shape[3]])
_c3 = self.diff_c3(pd.concat((_c3_1, _c3_2), axis=1)) + \
F.interpolate(_c4, scale_factor=2, mode="bilinear")
p_c3 = self.make_pred_c3(_c3)
outputs.append(p_c3)
_c3_up = resize(_c3,
size=c1_2.shape[2:],
mode='bilinear',
align_corners=False)
# Stage 2: x1/8 scale
_c2_1 = self.linear_c2(c2_1).transpose([0, 2, 1]).reshape(
[n, -1, c2_1.shape[2], c2_1.shape[3]])
_c2_2 = self.linear_c2(c2_2).transpose([0, 2, 1]).reshape(
[n, -1, c2_2.shape[2], c2_2.shape[3]])
_c2 = self.diff_c2(pd.concat((_c2_1, _c2_2), axis=1)) + \
F.interpolate(_c3, scale_factor=2, mode="bilinear")
p_c2 = self.make_pred_c2(_c2)
outputs.append(p_c2)
_c2_up = resize(_c2,
size=c1_2.shape[2:],
mode='bilinear',
align_corners=False)
# Stage 1: x1/4 scale
_c1_1 = self.linear_c1(c1_1).transpose([0, 2, 1]).reshape(
[n, -1, c1_1.shape[2], c1_1.shape[3]])
_c1_2 = self.linear_c1(c1_2).transpose([0, 2, 1]).reshape(
[n, -1, c1_2.shape[2], c1_2.shape[3]])
_c1 = self.diff_c1(pd.concat((_c1_1, _c1_2), axis=1)) + \
F.interpolate(_c2, scale_factor=2, mode="bilinear")
p_c1 = self.make_pred_c1(_c1)
outputs.append(p_c1)
# Linear Fusion of difference image from all scales
_c = self.linear_fuse(pd.concat((_c4_up, _c3_up, _c2_up, _c1), axis=1))
# #Dropout
# if dropout_ratio > 0:
# self.dropout = nn.Dropout2d(dropout_ratio)
# else:
# self.dropout = None
# Upsampling x2 (x1/2 scale)
x = self.convd2x(_c)
# Residual block
x = self.dense_2x(x)
# Upsampling x2 (x1 scale)
x = self.convd1x(x)
# Residual block
x = self.dense_1x(x)
# Final prediction
cp = self.change_probability(x)
outputs.append(cp)
if self.output_softmax:
temp = outputs
outputs = []
for pred in temp:
outputs.append(self.active(pred))
return outputs
（三）复现精度
在LEVIR-CD测试集的测试效果如下，通过验收。
| Network | OPT | Epoch | Batch-Size | Dataset | F1-Score |
|---|---|---|---|---|---|
| ChangeFormer | AdamW | 200 | 16 | LEVIR-CD | 90.347% |
（四） 环境和数据准备
- 克隆仓库：
%cd work
!git clone https://github.com/HULEIYI/ChangeFormer-pd.git
# 注：代码已经放在work目录下并配置好了路径，建议直接使用。
正克隆到 'ChangeFormer-pd'...
remote: Enumerating objects: 182, done.[K
remote: Counting objects: 100% (182/182), done.[K
remote: Compressing objects: 100% (156/156), done.[K
remote: Total 182 (delta 26), reused 168 (delta 19), pack-reused 0[K
接收对象中: 100% (182/182), 32.78 MiB | 27.00 KiB/s, 完成.
处理 delta 中: 100% (26/26), 完成.
检查连接... 完成。
- 解压数据和预训练权重：
!unzip -qo data/data161372/Data-and-PreWeight.zip -d data/data161372
- 解压训练好的权重：
!unzip -qo data/data162790/checkpoints.zip -d data/data162790
!unzip -qo data/data162790/pretrained_changeformer.zip -d data/data162790
%mv data/data162790/checkpoints/CD_ChangeFormerV6_LEVIR_b16_lr0.0001_adamw_trainval_test_200_linear_ce_multi_train_True_multi_infer_False_shuffle_AB_False_embed_dim_256/best_ckpt.pdparam data/data162790/checkpoints/CD_ChangeFormerV6_LEVIR_b16_lr0.0001_adamw_trainval_test_200_linear_ce_multi_train_True_multi_infer_False_shuffle_AB_False_embed_dim_256/best_ckpt.pdparams
%mv data/data162790/pretrained_changeformer/pretrained_changeformer.pdparam data/data162790/pretrained_changeformer/pretrained_changeformer.pdparams
- 安装依赖：
!pip install tifffile
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting tifffile
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d8/38/85ae5ed77598ca90558c17a2f79ddaba33173b31cf8d8f545d34d9134f0d/tifffile-2021.11.2-py3-none-any.whl (178 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m178.9/178.9 kB[0m [31m3.3 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hRequirement already satisfied: numpy>=1.15.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from tifffile) (1.19.5)
Installing collected packages: tifffile
Successfully installed tifffile-2021.11.2
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip available: [0m[31;49m22.1.2[0m[39;49m -> [0m[32;49m22.2.2[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpip install --upgrade pip[0m
（五）快速体验
- 模型训练：
- 使用
run_ChangeFormer_LEVIR.sh脚本即可快速开始训练，超参数已在脚本中对齐原文。
- 使用
%cd work/ChangeFormer-pd/scripts
/home/aistudio/work/ChangeFormer-pd/scripts
!sh run_ChangeFormer_LEVIR.sh
# 注意：
# 如果报错pretrained_changeformer.pdparams不存在，请在相应路径下将pretrained_changeformer.pdparam重命名为pretrained_changeformer.pdparams即可。
# 这一错误是上传数据集时命名错误，不影响训练。
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Sized
['gpu:0']
W0817 16:58:56.233995 960 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 10.1
W0817 16:58:56.238157 960 gpu_resources.cc:91] device: 0, cuDNN Version: 7.6.
initialize network with normal
================ (Wed Aug 17 16:58:57 2022) ================
gpu_ids: ['gpu:0'] project_name: CD_ChangeFormerV6_LEVIR_b16_lr0.0001_adamw_trainval_test_200_linear_ce_multi_train_True_multi_infer_False_shuffle_AB_False_embed_dim_256 checkpoint_root: ./checkpoints vis_root: ./vis num_workers: 0 dataset: CDDataset data_name: LEVIR batch_size: 16 split: trainval split_val: test img_size: 256 shuffle_AB: False n_class: 2 embed_dim: 256 pretrain: ../../data/data162790/pretrained_changeformer/pretrained_changeformer.pdparam multi_scale_train: True multi_scale_infer: False multi_pred_weights: [0.5, 0.5, 0.5, 0.8, 1.0] net_G: ChangeFormerV6 loss: ce optimizer: adamw lr: 0.0001 max_epochs: 200 lr_policy: linear lr_decay_iters: 100 checkpoint_dir: ./checkpoints/CD_ChangeFormerV6_LEVIR_b16_lr0.0001_adamw_trainval_test_200_linear_ce_multi_train_True_multi_infer_False_shuffle_AB_False_embed_dim_256 vis_dir: ./vis/CD_ChangeFormerV6_LEVIR_b16_lr0.0001_adamw_trainval_test_200_linear_ce_multi_train_True_multi_infer_False_shuffle_AB_False_embed_dim_256
Initializing backbone weights from: ../../data/data162790/pretrained_changeformer/pretrained_changeformer.pdparam
lr: 0.0001000
0%| | 0/509 [00:00<?, ?it/s]/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/norm.py:654: UserWarning: When training, we now always track global mean and variance.
"When training, we now always track global mean and variance.")
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/math_op_patch.py:278: UserWarning: The dtype of left and right variables are not the same, left dtype is paddle.float32, but right dtype is paddle.bool, the right dtype will convert to paddle.float32
format(lhs_dtype, rhs_dtype, lhs_dtype))
0%| | 1/509 [00:03<31:52, 3.77s/it]Is_training: True. [0,199][1,509], imps: 92.96, est: 77.87h, G_loss: 1.34087, running_mf1: 0.52390
8%|███▌ | 43/509 [01:10<12:27, 1.60s/it]^C
- 模型验证：
- 训练好的权重文件已放在
data/data162790/checkpoints下，可自行查看。 - 使用提供的
eval_ChangeFormer_LEVIR.sh文件即可体验。
- 训练好的权重文件已放在
!sh eval_ChangeFormer_LEVIR.sh
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Sized
[]
W0817 17:00:43.274778 1190 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 10.1
W0817 17:00:43.278940 1190 gpu_resources.cc:91] device: 0, cuDNN Version: 7.6.
initialize network with normal
gpu:0
================ (Wed Aug 17 17:00:44 2022) ================
gpu_ids: [] project_name: CD_ChangeFormerV6_LEVIR_b16_lr0.0001_adamw_trainval_test_200_linear_ce_multi_train_True_multi_infer_False_shuffle_AB_False_embed_dim_256 print_models: False checkpoints_root: ../../data/data162790/checkpoints vis_root: ./vis num_workers: 8 dataset: CDDataset data_name: LEVIR batch_size: 1 split: test img_size: 256 n_class: 2 embed_dim: 256 net_G: ChangeFormerV6 checkpoint_name: best_ckpt.pdparam checkpoint_dir: ../../data/data162790/checkpoints/CD_ChangeFormerV6_LEVIR_b16_lr0.0001_adamw_trainval_test_200_linear_ce_multi_train_True_multi_infer_False_shuffle_AB_False_embed_dim_256 vis_dir: ./vis/CD_ChangeFormerV6_LEVIR_b16_lr0.0001_adamw_trainval_test_200_linear_ce_multi_train_True_multi_infer_False_shuffle_AB_False_embed_dim_256 loading last checkpoint...
Eval Historical_best_acc = 0.9492 (at epoch 183)
Begin evaluation...
Is_training: False. [1,2048], running_mf1: 0.96424
Is_training: False. [101,2048], running_mf1: 0.50000
Is_training: False. [201,2048], running_mf1: 0.96965
Is_training: False. [301,2048], running_mf1: 0.98952
Is_training: False. [401,2048], running_mf1: 0.50000
Is_training: False. [501,2048], running_mf1: 0.49922
Is_training: False. [601,2048], running_mf1: 0.50000
Is_training: False. [701,2048], running_mf1: 0.88527
Is_training: False. [801,2048], running_mf1: 0.93557
Is_training: False. [901,2048], running_mf1: 0.89550
Is_training: False. [1001,2048], running_mf1: 0.97242
Is_training: False. [1101,2048], running_mf1: 0.92867
Is_training: False. [1201,2048], running_mf1: 0.49982
Is_training: False. [1301,2048], running_mf1: 0.95010
Is_training: False. [1401,2048], running_mf1: 0.50000
Is_training: False. [1501,2048], running_mf1: 0.96817
Is_training: False. [1601,2048], running_mf1: 0.97160
Is_training: False. [1701,2048], running_mf1: 0.49924
Is_training: False. [1801,2048], running_mf1: 0.50000
Is_training: False. [1901,2048], running_mf1: 0.49871
Is_training: False. [2001,2048], running_mf1: 0.96150
acc: 0.99034 miou: 0.90691 mf1: 0.94919 iou_0: 0.98988 iou_1: 0.82394 F1_0: 0.99491 F1_1: 0.90347 precision_0: 0.99399 precision_1: 0.91975 recall_0: 0.99584 recall_1: 0.88777
- 模型预测：
- 加载训练好的权重，使用模型进行预测。
- 使用
demo.py。- 生成预测图片在
ChangeFormer-pd/samples_LEVIR/predict_CD_ChangeFormerV6路径下。
- 生成预测图片在
%cd ..
/home/aistudio/work/ChangeFormer-pd
!python demo_LEVIR.py
W0817 18:22:09.385291 4511 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 10.1
W0817 18:22:09.389484 4511 gpu_resources.cc:91] device: 0, cuDNN Version: 7.6.
initialize network with normal
gpu:0
best_acc: 0.9491929514314832
process: ['test_77_0512_0256.png']
process: ['test_102_0512_0000.png']
process: ['test_121_0768_0256.png']
process: ['test_2_0000_0000.png']
process: ['test_2_0000_0512.png']
process: ['test_7_0256_0512.png']
process: ['test_55_0256_0000.png']
- 模型导出：
- 参数
model_path为.pdparams后缀的权重的路径 - 参数
save_inference_dir为静态图的保存文件夹路径
- 参数
!python export_model.py --model_path ../../data/data162790/checkpoints/CD_ChangeFormerV6_LEVIR_b16_lr0.0001_adamw_trainval_test_200_linear_ce_multi_train_True_multi_infer_False_shuffle_AB_False_embed_dim_256/best_ckpt.pdparams --save_inference_dir ./inference/
W0817 18:27:48.933506 5191 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 10.1
W0817 18:27:48.937757 5191 gpu_resources.cc:91] device: 0, cuDNN Version: 7.6.
- 静态图推理：
- 使用
infer.py文件进行推理。 - 参数
model_dir为静态图保存的文件夹路径 - 参数
img_dir为待预测的图像文件夹路径，文件夹下需包含A和B两个子文件夹
- 使用
!python infer.py --model_dir ./inference/ --img_dir ./samples_LEVIR/
total file number is 11
finish
（六）TIPC基础链条测试
该部分依赖auto_log，需要进行安装，安装方式如下：
auto_log的详细介绍参考https://github.com/LDOUBLEV/AutoLog。
#%cd ~
%cd /home/aistudio/work/ChangeFormer-pd
!git clone https://github.com/LDOUBLEV/AutoLog
%cd ./AutoLog/
!pip3 install -r requirements.txt
!python3 setup.py bdist_wheel
!pip3 install ./dist/auto_log-1.2.0-py3-none-any.whl
%cd ..
## 网络原因可能会失败。
- 运行命令，准备小批量数据
%cd work/ChangeFormer-pd/
!bash ./test_tipc/prepare.sh test_tipc/configs/ChangeFormer/train_infer_python.txt 'lite_train_lite_infer'
- 运行命令，小批量数据训练、导出、推理一体化：
!bash test_tipc/test_train_inference_python.sh test_tipc/configs/ChangeFormer/train_infer_python.txt 'lite_train_lite_infer'
（七）项目总结
- 项目基于论文原文以及【论文笔记】A Transformer-based Siamese network for change detection论文笔记对ChangeFormer进行了较为详细的说明。
- 项目给出了基于paddl框架的复现仓库地址，并基于代码给出了训练、测试和预测，以及ChangeFormer模型导出的快速体验命令。
- 关于模型代码部分，由于paddle和pytorch框架架构类似，其代码转换并不复杂，具体可以参考《论文复现赛指南》，其中提供了很多torch和paddle的api转换对，可以直接使用，对于paddle没有实现的api也提供了一些组合解决的方案。
- 对于没有解决方案的torch api，我的一点心得是查看torch的源码，将对应功能的源码改写成paddle格式一般就可以使用。
-另外，对于自己没有办法解决的问题，也可以在paddle的社区内进行求助，复现赛专属的rd也会帮助解决很多问题。这一点必须给我们paddle点赞！👍
- 对于没有解决方案的torch api，我的一点心得是查看torch的源码，将对应功能的源码改写成paddle格式一般就可以使用。
（八）致谢
- 感谢百度飞桨提供了论文复现赛这样的平台，在比赛冲奖的同时也提升了自己的深度学习代码能力，加深了对科研方向的理解。
- 特别感谢遥感复现组的rd晖哥在任务代码方面的帮助，以及芋泥啵啵姐的算力支持！
- 最最最后，感谢
孙孔远杭:KKKloveqbh同学的鼎力相助！
此文章为搬运
原项目链接
学大模型，用大模型上飞桨星河社区！每天8点V100G算力免费领！免费领取ERNIE 4.0 100w Token >>>
更多推荐
1
0- 0
已为社区贡献1438条内容
所有评论(0)