【OVERLORD】使用Paddle实现MRI医学图像超分辨率项目

• 相关项目1：【OVERLORD】IXISR医学图像超分数据集读取实践
• 相关项目2：

一、项目背景

1、核磁共振图像（MRI）

       核磁共振成像（Magnetic Resonance Imaging， MRI）是一种非侵入式的活体成像技术，科学将其定义为处于静磁场中的原子核在另一交变电磁场作用下发生的物理现象。通常人们所说的核磁共振就是利用核磁共振现象获取分子结构，以及人体内部结构信息的技术。共振成像的基本原理，是将人体置于特殊的磁场中，用无线电射频脉冲激发人体内氢原子核，引起氢原子核共振并吸收能量，在停止射频脉冲后，氢原子核按特定频率发出射电信号，并将吸收的能量释放出来，被体外的接受器收录，经电子计算机处理获得图像。

       核磁共振成像技术提供的信息量不但大于医学影像学中的其他许多成像术，而且不同于已有的成像术。因此，它对疾病的诊断具有很大的潜在优越性。它可以直接作出横断面、矢状面、冠状面和各种斜面的体层图像，并且不会产生CT检测中的伪影;不需注射造影剂;无电离辐射，对机体不良影响较小。共振成像对检测脑内血肿、脑外血肿、脑肿瘤、颅内动脉瘤、动静脉血管畸形、脑缺血、椎管内肿瘤、脊髓空洞症和脊髓积水等疾病非常有效。同时，对腰椎椎间盘后突、原发性肝癌等疾病的诊断也很有效。

       当然，共振成像也存在不足之处。它的速度相对较慢，空间分辨率不及CT，且存在运动伪影等。图像重建和超分辨率是MRI中的两项关键技术。前者通过降低空间采样率来加速MRI（但其中涉及到下采样伪影，运动伪影等等处理）相关算法比如并行成像技术，压缩感知成像等等，后者通过恢复单个退化的低分辨率（LR）图像来实现高分辨率（HR）图像，相关算法比如双三次插值等等。

2、核磁共振图像超分辨率重建

       高分辨率磁共振 (MR) 成像在许多临床应用中是可取的，因为它的体素尺寸更小，可以为医生提供更加精确的结构和纹理细节，有助于更准确的后续分析和早期临床诊断。然而高分辨率核磁共振图像的生成往往受到许多因素影响，例如硬件设备、成像时间、人身体的运动、环境噪声的影响等等。这些影响因素往往是混合出现的，因此无法只针对其一进行优化。因此为了将磁共振获得到的低分辨率图像进行有效的高分辨率还原，图像超分辨率是一种有效且具有成本效益的优秀技术，可提高 MR 图像的空间分辨率。此技术为低分辨率MRI图像的高信噪比和高分辨率重建提供了可行性。

3、医学图像超分辨率技术的发展

       传统的超分辨率算法包括基于插值、基于重建以及基于样例的方法,这些方法普遍难以重建得到图像的高频细节信息,计算较为复杂，重建所需时间较长，并且无法满足X4等尺度的需求。为了解决这些问题，近年来学者们将深度学习应用于超分辨率重建中，并取得了很多的突破，如今，基于深度学习的超分辨率算法已经占据了超分辨率算法研究的主流位置。在医学图像领域中，基于深度学习的超分辨率算法可以从医学影像训练集数据中获取先验知识，并根据这些信息，通过神经网络将低分辨率图像重建为高分辨率图像。

       近年来，深度学习尤其是深度卷积神经网络（Convolutional Neural Network，CNN）[1-3] 在MRI重建领域得到了广泛的应用。基于深度学习的MRI成像方法在保证图像精度的同时提高重建速度。Dong[4] 使用超分辨率卷积神经网络（Super-Resolution Convolutional Neural Network， SRCNN）进行图像重建，具有结构简单、容易实现等优点。由于存在采用的卷积层数少、感受野小、泛化能力差等缺陷，使SRCNN因无法提取图像深层次特征而导致重建图像纹理不够清晰。基于参考图像的超分辨率重建（Reference-based Super-Resolution，RefSR）技术在图像重建过程中，通过引入与低分辨率图像具有相似纹理或内容结构的参考图像，将参考图像的高频细节迁移到低分辨率图像（low-resolution，LR）中，从而获得高分辨率图像。

二、项目难点

• 【清晰重建MRI图像细节与边缘】
• 【提升重建MRI图像的高频细节丰富度】
• 【有效减少伪影及噪声】

三、项目方案及数据说明

为了获取更高分辨率的MR图像，我们使用此数据集，和5个简单的模型进行超分辨率重建。

• 数据集及数据加载方法详情：【OVERLORD】IXISR医学图像超分数据集读取实践

1、项目使用的数据集

       IXISR dataset[5]是作者Zhao X通过对IXI dataset[6] 进行进一步处理从而构建的。原IXI数据包含三种类型的 MR 图像：581 个 T1 数据、578 个 T2 数据和 578 个 PD 数据。作者取这三个子集的交集，为每种类型的 MR 图像生成 576 个 3D 体积。然后将这些 3D 体积裁剪为 240×240×96（高×宽×深）的大小，以适应 3 个缩放因子（×2、×3 和 ×4）。

• HR图像：为原图像数据。
• LR图像：根据双三次下采样和k空间截断（HR 图像首先通过离散傅里叶变换 (DFT) 转换为 k 空间，然后沿高度和宽度方向进行截断）生成的。

       IXISR数据集的维度如下：

              训练集LR图像：(500, 120, 120, 96)

              训练集HR图像：(500, 240, 240, 96)

              验证集LR图像：(6, 120, 120, 96)

              验证集HR图像：(6, 240, 240, 96)

              测试集LR图像：(70, 120, 120, 96)

              测试集HR图像：(70, 240, 240, 96)

2、项目使用的模型

• 模型及模型结构详情：【OVERLORD】IXISR医学图像超分数据集读取实践

Module	Year
Sub-Pixel	2016

Module	Year
VRCNN	2017

Module	Year
EDSR	2017

Module	Year
SRResNet	2017

3、其他方法

大家也可以直接使用PaddleHub的预训练模型

Module	PaddleHub link
DCSCN	https://www.paddlepaddle.org.cn/hubdetail?name=dcscn

Module	PaddleHub link
DCSCN	https://www.paddlepaddle.org.cn/hubdetail?name=dcscn
FALSR_A	https://www.paddlepaddle.org.cn/hubdetail?name=falsr_a
FALSR_B	https://www.paddlepaddle.org.cn/hubdetail?name=falsr_b
FALSR_C	https://www.paddlepaddle.org.cn/hubdetail?name=falsr_c

大家还可以直接使用PaddleGAN

Module
RealSR
ESRGAN
LESRCNN
PAN
DRN

四、项目过程

1、解压数据集

# 1. 由于压缩包分包了，所以我们需要将它放在一个压缩包内，随后再解压 -q 执行时不显示任何信息
!zip -q -s 0  ./data/data155555/IXISR.zip  --out ./data/data155555/IXI.zip
# 2. 随后解压IXISR数据集
!unzip -q -o ./data/data155555/IXI.zip -d ./data/data155555/
!echo "unzip finish!!"

unzip finish!!

2、安装并导入包

# 安装必要的包
!pip install scikit-image -U -i https://pypi.tuna.tsinghua.edu.cn/simple

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: scikit-image in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (0.19.3)
Requirement already satisfied: networkx>=2.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image) (2.4)
Requirement already satisfied: imageio>=2.4.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image) (2.6.1)
Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image) (21.3)
Requirement already satisfied: pillow!=7.1.0,!=7.1.1,!=8.3.0,>=6.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image) (8.2.0)
Requirement already satisfied: PyWavelets>=1.1.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image) (1.3.0)
Requirement already satisfied: numpy>=1.17.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image) (1.19.5)
Requirement already satisfied: scipy>=1.4.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image) (1.6.3)
Requirement already satisfied: tifffile>=2019.7.26 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image) (2021.11.2)
Requirement already satisfied: decorator>=4.3.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from networkx>=2.2->scikit-image) (4.4.2)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from packaging>=20.0->scikit-image) (3.0.9)

# 导入包
import math
import skimage
import numpy as np
import matplotlib.pyplot as plt
import os
from PIL import Image
import paddle
import paddle as P
import paddle.nn as nn
import paddle.nn.functional as F
from paddle import ParamAttr
from paddle.distributed import ParallelEnv
from paddle.nn import Conv2D, BatchNorm, Linear, Dropout, AdaptiveAvgPool2D, MaxPool2D, AvgPool2D

nn.initializer.set_global_initializer(nn.initializer.Normal(mean=0.0,std=0.01), nn.initializer.Constant())

3、超参数及路径配置

# 3. 设置一下各种文件地址

data_mode_index = 3
data_type_index = 2

data_mode = ['bicubic_2x','bicubic_3x','bicubic_4x','truncation_2x','truncation_3x','truncation_4x']
data_type = ['PD','T1','T2']

SCALE = 2
EPOCH = 10

# ./data/data155555/IXI/IXI_LR/bicubic_2x/PD/train

train_LR_path = './data/data155555/IXI/IXI_LR/{}/{}/train'.format(data_mode[data_mode_index],data_type[data_type_index])
train_HR_path = './data/data155555/IXI/IXI_HR/{}/train'.format(data_type[data_type_index])

valid_LR_path = './data/data155555/IXI/IXI_LR/{}/{}/valid'.format(data_mode[data_mode_index],data_type[data_type_index])
valid_HR_path = './data/data155555/IXI/IXI_HR/{}/valid'.format(data_type[data_type_index])

test_LR_path = './data/data155555/IXI/IXI_LR/{}/{}/test'.format(data_mode[data_mode_index],data_type[data_type_index])
test_HR_path = './data/data155555/IXI/IXI_HR/{}/test'.format(data_type[data_type_index])

4、加载数据集

# 加载所有的npy文件
from glob import glob
import os
import numpy as np

train_LR_paths = sorted(glob(os.path.join(train_LR_path,"*_*.npy")))
train_LR_data = np.array([np.load(fname) for fname in train_LR_paths], dtype='float32')

train_HR_paths = sorted(glob(os.path.join(train_HR_path,"*_*.npy")))
# print(train_HR_paths)
train_HR_data = np.array([np.load(fname) for fname in train_HR_paths], dtype='float32')

valid_LR_paths = sorted(glob(os.path.join(valid_LR_path,"*_*.npy")))
valid_LR_data = np.array([np.load(fname) for fname in valid_LR_paths], dtype='float32')

valid_HR_paths = sorted(glob(os.path.join(valid_HR_path,"*_*.npy")))
valid_HR_data = np.array([np.load(fname) for fname in valid_HR_paths], dtype='float32')

test_LR_paths = sorted(glob(os.path.join(test_LR_path,"*_*.npy")))
test_LR_data = np.array([np.load(fname) for fname in test_LR_paths], dtype='float32')

test_HR_paths = sorted(glob(os.path.join(test_HR_path,"*_*.npy")))
test_HR_data = np.array([np.load(fname) for fname in test_HR_paths], dtype='float32')

# 查看数据维度为多少

print(train_LR_data.shape)
print(train_HR_data.shape)
print(valid_LR_data.shape)
print(valid_HR_data.shape)
print(test_LR_data.shape)
print(test_HR_data.shape)

(500, 120, 120, 96)
(500, 240, 240, 96)
(6, 120, 120, 96)
(6, 240, 240, 96)
(70, 120, 120, 96)
(70, 240, 240, 96)

5、数据转化2D并可视化

 #将3D切片数据集转化为2D切片数据方便训练

train_LR_data = train_LR_data.transpose([0,3,1,2])
train_HR_data = train_HR_data.transpose([0,3,1,2])
valid_LR_data = valid_LR_data.transpose([0,3,1,2])
valid_HR_data = valid_HR_data.transpose([0,3,1,2])
test_LR_data  = test_LR_data .transpose([0,3,1,2])
test_HR_data  = test_HR_data .transpose([0,3,1,2])

train_LR_data = train_LR_data.reshape(-1,train_LR_data.shape[2],train_LR_data.shape[3])
train_HR_data = train_HR_data.reshape(-1,train_HR_data.shape[2],train_HR_data.shape[3])
valid_LR_data = valid_LR_data.reshape(-1,valid_LR_data.shape[2],valid_LR_data.shape[3])
valid_HR_data = valid_HR_data.reshape(-1,valid_HR_data.shape[2],valid_HR_data.shape[3])
test_LR_data  = test_LR_data .reshape(-1,test_LR_data .shape[2],test_LR_data .shape[3])
test_HR_data  = test_HR_data .reshape(-1,test_HR_data .shape[2],test_HR_data .shape[3])

print(train_LR_data.shape)
print(train_HR_data.shape)
print(valid_LR_data.shape)
print(valid_HR_data.shape)
print(test_LR_data.shape)
print(test_HR_data.shape)

(48000, 120, 120)
(48000, 240, 240)
(576, 120, 120)
(576, 240, 240)
(6720, 120, 120)
(6720, 240, 240)

# 查看MRI图像
import matplotlib.pyplot as plt

aa = train_LR_data[0,:,:]
bb = train_HR_data[0,:,:]

# aa = np.where(aa>255.,255.,aa)
# bb = np.where(bb>255.,255.,bb)

aa = (aa-aa.min())/(aa.max()-aa.min())
bb = (bb-bb.min())/(bb.max()-bb.min())

fig = plt.figure(figsize=(10,10))
plt.subplot(2,3,1)
plt.imshow(aa,cmap='gray')
plt.subplot(2,3,2)
plt.imshow(bb,cmap='gray')


print(aa.min(),aa.max())
print(aa.min(),bb.max())

plt.show()

0.0 1.0
0.0 1.0


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2349: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  if isinstance(obj, collections.Iterator):
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2366: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  return list(data) if isinstance(data, collections.MappingView) else data
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/image.py:425: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  a_min = np.asscalar(a_min.astype(scaled_dtype))
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/image.py:426: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  a_max = np.asscalar(a_max.astype(scaled_dtype))

6、自定义数据加载类 MyDataset

# 没有进行patch分割成小块训练的
from paddle.io import Dataset  # 导入类库 Dataset
from paddle.vision.transforms import ToTensor

class MyDataset(Dataset):  # 定义Dataset的子类MyDataset
    def __init__(self, mode='train',transform=ToTensor()):
        super(MyDataset, self).__init__()
        self.mode = mode
        if mode == 'train':
            self.LR_data = train_LR_data
            self.HR_data = train_HR_data
        elif mode == 'valid':
            self.LR_data = valid_LR_data
            self.HR_data = valid_HR_data
        else:
            self.LR_data = test_LR_data
            self.HR_data = test_HR_data

        self.transform = transform

    def data_augmentation(self,LR, HR): # 数据增强：随机翻转、旋转
        if np.random.randint(2) == 1:
            LR = LR[:,::-1]
            HR = HR[:,::-1]
        n = np.random.randint(4)
        if n == 1:
            LR = LR[::-1,:].transpose([1,0])
            HR = HR[::-1,:].transpose([1,0])
        if n == 2:
            LR = LR[::-1,::-1]
            HR = HR[::-1,::-1]
        if n == 3:
            LR = LR[:,::-1].transpose([1,0])
            HR = HR[:,::-1].transpose([1,0])
        return LR, HR

    def __getitem__(self, index):
        LR = self.LR_data[index]
        HR = self.HR_data[index]

        LR = (LR-LR.min())/(LR.max()-LR.min())
        HR = (HR-HR.min())/(HR.max()-HR.min())

        # 修改归一化

        if(self.mode=='train'):
            LR,HR = self.data_augmentation(LR,HR)
        LR = self.transform(LR)
        HR = self.transform(HR)
        return LR, HR

    def __len__(self):
        return self.LR_data.shape[0]

if __name__ == '__main__':
    import cv2
    from tqdm import tqdm_notebook
    test_dataset = MyDataset(mode='test')

    psnr_list = []
    ssim_list = []

    psnr = PSNR()
    ssim = SSIM()

    for i in tqdm_notebook(range(len(test_dataset))):

        lr,hr = test_dataset[i]

        hrr = lr[0,:,:].numpy()
        
        hrr = cv2.resize(hrr,(hrr.shape[0]*2,hrr.shape[1]*2),interpolation=cv2.INTER_CUBIC)

        hrr = paddle.Tensor(hrr)

        ssim.update(hrr.unsqueeze(0).unsqueeze(0),hr.unsqueeze(0))

        psnr.update(hrr.unsqueeze(0).unsqueeze(0),hr.unsqueeze(0))

        # psnr_list.append(psnr.psnr)
        # ssim_list.append(ssim.ssim)

        # plt.figure(figsize=(10,10))
        # plt.subplot(1,3,1)
        # plt.imshow(lr[0],cmap='gray')
        # plt.subplot(1,3,2)
        # plt.imshow(hr[0],cmap='gray')
        # plt.subplot(1,3,3)
        # plt.imshow(hrr,cmap='gray')
        # plt.show()

    print(psnr.accumulate())
    print(ssim.accumulate())

    # psnr_list = np.array(psnr_list)
    # ssim_list = np.array(ssim_list)

    # print(psnr_list.mean())
    # print(ssim_list.mean())

    # 0,0
    # 27.589457979556325
    # 0.9547699995465391

    # 0,1
    # 27.135037800861937
    # 0.9390757194948173

    # 0,2
    # 30.20191546584343
    # 0.9500627624909374

    # 3,0
    # 27.35476789852292
    # 0.9360441472609669

    # 3,1
    # 26.844071023862334
    # 0.9063720558462729

    # 3,2
    # 30.059611559149367
    # 0.9324317678299852

HBox(children=(IntProgress(value=0, max=6720), HTML(value='')))


30.059611559149285
0.9324317678299868

# 进行patch分割成小块训练的
from paddle.io import Dataset  # 导入类库 Dataset
from paddle.vision.transforms import ToTensor

class MyDataset(Dataset):  # 定义Dataset的子类MyDataset
    def __init__(self, mode='train',transform=ToTensor(),scale=2,patchsize=[24,24]):
        super(MyDataset, self).__init__()
        self.scale = scale
        self.patchsize = patchsize
        self.mode = mode
        if mode == 'train':
            self.LR_data = train_LR_data
            self.HR_data = train_HR_data
        elif mode == 'valid':
            self.LR_data = valid_LR_data
            self.HR_data = valid_HR_data
        else:
            self.LR_data = test_LR_data
            self.HR_data = test_HR_data

        self.transform = transform

    def data_augmentation(self,LR, HR): #数据增强：随机翻转、旋转
        if np.random.randint(2) == 1:
            LR = LR[:,::-1]
            HR = HR[:,::-1]
        n = np.random.randint(4)
        if n == 1:
            LR = LR[::-1,:].transpose([1,0])
            HR = HR[::-1,:].transpose([1,0])
        if n == 2:
            LR = LR[::-1,::-1]
            HR = HR[::-1,::-1]
        if n == 3:
            LR = LR[:,::-1].transpose([1,0])
            HR = HR[:,::-1].transpose([1,0])
        return LR, HR

    def getPatch(self,LR, HR): 
        sz = LR.shape
        ##对图片进行裁剪
        sz_row = sz[1] // (self.patchsize[0]*self.scale) * self.patchsize[0]*self.scale
        sz_col = sz[0] // (self.patchsize[1]*self.scale) * self.patchsize[1]*self.scale
        dif_row = sz[1] - sz_row
        dif_col = sz[0] - sz_col
        row_min = np.random.randint(dif_row + 1)    #设置随机裁剪行的起点
        col_min = np.random.randint(dif_col + 1)    #设置随机裁剪列的起点

        LRP = LR[row_min:row_min+self.patchsize[0],col_min:col_min+self.patchsize[1]]
        HRP = HR[row_min*self.scale:(row_min+self.patchsize[0])*self.scale,col_min*self.scale:(col_min+self.patchsize[1])*self.scale]

        return LRP,HRP
    def __getitem__(self, index):
        LR = self.LR_data[index]
        HR = self.HR_data[index]

        LR = (LR-LR.min())/(LR.max()-LR.min())
        HR = (HR-HR.min())/(HR.max()-HR.min())
        
        if(self.mode=='train'):
            LR,HR = self.getPatch(LR,HR)
            LR,HR = self.data_augmentation(LR,HR)

        LR = self.transform(LR)
        HR = self.transform(HR)
        return LR , HR

    def __len__(self):
        return self.LR_data.shape[0]

if __name__ == '__main__':
    test_dataset = MyDataset(mode='train')
    lr,hr = test_dataset[0]

    print(np.array(lr).min(),np.array(lr).max())
    print(np.array(hr).min(),np.array(hr).max())

    plt.subplot(1,2,1)
    plt.imshow(lr[0],cmap='gray')
    plt.subplot(1,2,2)
    plt.imshow(hr[0],cmap='gray')
    plt.show()

0.0 0.8236715
0.0 0.897937

7、自定义评估函数 MyMetrics

from paddle.metric import Metric


##########################################
#
#         PSNR 评估指标（峰值信噪比）
#
##########################################
class PSNR(Metric):

    def __init__(self, name='psnr', *args, **kwargs):
        super(PSNR, self).__init__(*args, **kwargs)
        self.psnr = 0
        self._name = name
        self.count = 0

    def _is_numpy_(self,var):
        return isinstance(var, (np.ndarray, np.generic))

    def update(self, preds, labels):

        if isinstance(preds, paddle.Tensor):
            # preds = paddle.nn.functional.sigmoid(preds)
            preds = preds.numpy()
        elif not self._is_numpy_(preds):
            raise ValueError("The 'preds' must be a numpy ndarray or Tensor.")

        if isinstance(labels, paddle.Tensor):
            labels = labels.numpy()
        elif not self._is_numpy_(labels):
            raise ValueError("The 'labels' must be a numpy ndarray or Tensor.")
        
        B,C,H,W = preds.shape[0],preds.shape[1],preds.shape[2],preds.shape[3]

        temp = 0
        for b in range(B):
            for c in range(C):
                temp += skimage.metrics.peak_signal_noise_ratio(preds[b,c,:,:], labels[b,c,:,:], data_range=1)

        temp = temp/(B*C)

        self.psnr += temp
        self.count += 1


    def reset(self):
        self.psnr = 0
        self.count = 0

    def accumulate(self):
        return self.psnr/self.count 

    def name(self):
        return self._name

##########################################
#
#        SSIM 评估指标（结构相似性）
#
##########################################
class SSIM(Metric):

    def __init__(self, name='ssim', *args, **kwargs):
        super(SSIM, self).__init__(*args, **kwargs)
        self.ssim = 0
        self._name = name
        self.count = 0

    def _is_numpy_(self,var):
        return isinstance(var, (np.ndarray, np.generic))

    def update(self, preds, labels):
        if isinstance(preds, paddle.Tensor):
            # preds = paddle.nn.functional.sigmoid(preds)
            preds = preds.numpy()
        elif not self._is_numpy_(preds):
            raise ValueError("The 'preds' must be a numpy ndarray or Tensor.")

        if isinstance(labels, paddle.Tensor):
            labels = labels.numpy()
        elif not self._is_numpy_(labels):
            raise ValueError("The 'labels' must be a numpy ndarray or Tensor.")

        B,C,H,W = preds.shape[0],preds.shape[1],preds.shape[2],preds.shape[3]

        temp = 0
        for b in range(B):
            for c in range(C):
                temp += skimage.metrics.structural_similarity(preds[b,c,:,:], labels[b,c,:,:], data_range=1)

        temp = temp/(B*C)

        self.ssim += temp

        self.count += 1

    def reset(self):
        self.ssim = 0
        self.count = 0

    def accumulate(self):
        return self.ssim/self.count 

    def name(self):
        return self._name

##########################################
#
#        MSE 评估指标（均方误差）
#
##########################################
class MSE(Metric):

    def __init__(self, name='mse', *args, **kwargs):
        super(MSE, self).__init__(*args, **kwargs)
        self.mse = 0
        self._name = name
        self.count = 0

    def _is_numpy_(self,var):
        return isinstance(var, (np.ndarray, np.generic))

    def update(self, preds, labels):
        if isinstance(preds, paddle.Tensor):
            # preds = paddle.nn.functional.sigmoid(preds)
            preds = preds.numpy()
        elif not self._is_numpy_(preds):
            raise ValueError("The 'preds' must be a numpy ndarray or Tensor.")

        if isinstance(labels, paddle.Tensor):
            labels = labels.numpy()
        elif not self._is_numpy_(labels):
            raise ValueError("The 'labels' must be a numpy ndarray or Tensor.")

        B,C,H,W = preds.shape[0],preds.shape[1],preds.shape[2],preds.shape[3]

        temp = 0
        for b in range(B):
            for c in range(C):
                temp += skimage.metrics.mean_squared_error(preds[b,c,:,:], labels[b,c,:,:])

        temp = temp/(B*C)

        self.mse += temp

        self.count += 1

    def reset(self):
        self.mse = 0
        self.count = 0

    def accumulate(self):
        return self.mse/self.count

    def name(self):
        return self._name

##########################################
#
#        RMSE 评估指标（均方根误差）
#
##########################################
class RMSE(Metric):

    def __init__(self, name='rmse', *args, **kwargs):
        super(RMSE, self).__init__(*args, **kwargs)
        self.rmse = 0
        self._name = name
        self.count = 0

    def _is_numpy_(self,var):
        return isinstance(var, (np.ndarray, np.generic))

    def update(self, preds, labels):
        if isinstance(preds, paddle.Tensor):
            # preds = paddle.nn.functional.sigmoid(preds)
            preds = preds.numpy()
        elif not self._is_numpy_(preds):
            raise ValueError("The 'preds' must be a numpy ndarray or Tensor.")

        if isinstance(labels, paddle.Tensor):
            labels = labels.numpy()
        elif not self._is_numpy_(labels):
            raise ValueError("The 'labels' must be a numpy ndarray or Tensor.")

        B,C,H,W = preds.shape[0],preds.shape[1],preds.shape[2],preds.shape[3]

        temp = 0
        for b in range(B):
            for c in range(C):
                temp += skimage.metrics.mean_squared_error(preds[b,c,:,:], labels[b,c,:,:])

        temp = temp/(B*C)

        temp = math.sqrt(temp)

        self.rmse += temp

        self.count += 1

    def reset(self):
        self.rmse = 0
        self.count = 0

    def accumulate(self):
        return self.rmse/self.count

    def name(self):
        return self._name

##########################################
#
#        NRMSE 评估指标（归一化均方根误差）
#
##########################################
class NRMSE(Metric):

    def __init__(self, name='nrmse', *args, **kwargs):
        super(NRMSE, self).__init__(*args, **kwargs)
        self.nrmse = 0
        self._name = name
        self.count = 0

    def _is_numpy_(self,var):
        return isinstance(var, (np.ndarray, np.generic))

    def update(self, preds, labels):
        if isinstance(preds, paddle.Tensor):
            # preds = paddle.nn.functional.sigmoid(preds)
            preds = preds.numpy()
        elif not self._is_numpy_(preds):
            raise ValueError("The 'preds' must be a numpy ndarray or Tensor.")

        if isinstance(labels, paddle.Tensor):
            labels = labels.numpy()
        elif not self._is_numpy_(labels):
            raise ValueError("The 'labels' must be a numpy ndarray or Tensor.")

        B,C,H,W = preds.shape[0],preds.shape[1],preds.shape[2],preds.shape[3]

        temp = 0
        for b in range(B):
            for c in range(C):
                temp += skimage.metrics.normalized_root_mse(preds[b,c,:,:], labels[b,c,:,:])

        temp = temp/(B*C)

        self.nrmse += temp

        self.count += 1

    def reset(self):
        self.nrmse = 0
        self.count = 0

    def accumulate(self):
        return self.nrmse/self.count

    def name(self):
        return self._name

8、自定义损失函数 MyLosses

# 超分常用的loss参考：https://zhuanlan.zhihu.com/p/366274586

import paddle
import paddle.nn as nn
import paddle.nn.functional as F
import numpy as np
from math import exp

#from work.models.VGG import vgg19

# 计算一维的高斯分布向量
def gaussian(window_size, sigma):
    gauss = paddle.to_tensor([exp(-(x - window_size/2)**2/float(2*sigma**2)) for x in range(window_size)])
    return gauss/gauss.sum()

# 创建高斯核，通过两个一维高斯分布向量进行矩阵乘法得到
def create_window(window_size, channel):
    _1D_window = gaussian(window_size, 1.5).unsqueeze(1)
    _2D_window = _1D_window.mm(_1D_window.t()).astype('float32').unsqueeze(0).unsqueeze(0)
    window = paddle.to_tensor(_2D_window.expand([channel, 1, window_size, window_size]))
    return window

# 计算SSIM
# 直接使用SSIM的公式，但是在计算均值时，不是直接求像素平均值，而是采用归一化的高斯核卷积来代替。
# 在计算方差和协方差时用到了公式Var(X)=E[X^2]-E[X]^2, cov(X,Y)=E[XY]-E[X]E[Y].
# 正如前面提到的，上面求期望的操作采用高斯核卷积代替
def _ssim(img1, img2, window, window_size, channel, size_average = True):
    img1= paddle.to_tensor(img1)
    img2= paddle.to_tensor(img2)
    mu1 = F.conv2d(img1, window, padding = window_size//2, groups = channel)
    mu2 = F.conv2d(img2, window, padding = window_size//2, groups = channel)

    mu1_sq = mu1.pow(2)
    mu2_sq = mu2.pow(2)
    mu1_mu2 = mu1*mu2

    sigma1_sq = F.conv2d(img1*img1, window, padding = window_size//2, groups = channel) - mu1_sq #Var(X)=E[X^2]-E[X]^2
    sigma2_sq = F.conv2d(img2*img2, window, padding = window_size//2, groups = channel) - mu2_sq
    sigma12 = F.conv2d(img1*img2, window, padding = window_size//2, groups = channel) - mu1_mu2  #cov(X,Y)=E[XY]-E[X]E[Y]

    C1 = 0.01**2
    C2 = 0.03**2

    ssim_map = ((2*mu1_mu2 + C1)*(2*sigma12 + C2))/((mu1_sq + mu2_sq + C1)*(sigma1_sq + sigma2_sq + C2))

    if size_average:
        return ssim_map.mean()
    else:
        return ssim_map.mean(1).mean(1).mean(1)
#类重用窗口
class SSIMLoss(nn.Layer):
    def __init__(self, window_size = 11, size_average = True):
        super(SSIMLoss, self).__init__()  #对继承自父类的属性进行初始化。而且是用父类的初始化方法来初始化继承的属性。
        self.window_size = window_size
        self.size_average = size_average
        self.channel = 1
        self.window = create_window(window_size, self.channel)
        self.l1=paddle.nn.L1Loss(reduction='mean')
    def forward(self, img1, img2):
        l1loss = self.l1(img1,img2)
        (_, channel, _, _) = img1.shape

        if channel == self.channel:
            window = self.window
        else:
            window = create_window(self.window_size, channel)
            self.window = window
            self.channel = channel

        return l1loss+0.2*(1 - _ssim(img1, img2, window, self.window_size, channel, self.size_average))

def ssim(img1, img2, window_size = 11, size_average = True):
    (_, channel, _, _) = img1.shape
    window = create_window(window_size, channel)
    return _ssim(img1, img2, window, window_size, channel, size_average)

def expandedSobel(inputTensor,sobelFilter):

    #this considers data_format = 'NCHW'
    inputChannels = paddle.reshape(paddle.ones_like(inputTensor[0,:,0,0]),(1,1,-1,1))

    return sobelFilter * inputChannels

class AdvSoberLoss(nn.Layer):

    def __init__(self, alpha=0.25):
        super(AdvSoberLoss, self).__init__()
        self.sobelFilter  = paddle.to_tensor([[[[0.,  2.]], [[-1.,  1.]],[[-2.,  0.]]],
                      [[[1.,  1.]], [[0.,  0.]],[[-1.,  -1.]]],
                      [[[2., 0.]], [[1., -1.]],[[0., -2.]]]], dtype='float32').reshape(
            (3,3, 1, 2))
        #print(self.sobelFilter)
        self.alpha=alpha
        self.l1_loss = paddle.nn.L1Loss(reduction='mean')

    def forward(self, logits1,label):

        label.stop_gradient = True
        filt = expandedSobel(label,self.sobelFilter)
        total_loss=0

        sobelLabel = F.conv2d(label,filt)
        sobelPred =  F.conv2d(logits1,filt)
        sobelLoss = paddle.mean(paddle.abs(sobelLabel - sobelPred))
        Loss= self.l1_loss(logits1,label)+self.alpha*sobelLoss
        total_loss=Loss

        return total_loss


from paddle.vision.models import vgg19

class LossVGG19(nn.Layer):
    def __init__(self):
        super(LossVGG19, self).__init__()

        self.vgg_layers = vgg19(pretrained=True).features
        self.layer_name_mapping = {
            '3': "relu1",
            '8': "relu2",
            '13': "relu3",
            '22': "relu4",
            '31': "relu5",  # 1_2 to 5_2
        }

    def forward(self, x):
        output = {}

        for name, module in self.vgg_layers._sub_layers.items():
            x = module(x)
            if name in self.layer_name_mapping:
                output[self.layer_name_mapping[name]] = x

        return output

        
def loss_textures(x, y, nc=3, alpha=1.2, margin=0):
    xi = x.reshape([x.shape[0], -1, nc, x.shape[2], x.shape[3]])
    yi = y.reshape([y.shape[0], -1, nc, y.shape[2], y.shape[3]])

    xi2 = paddle.sum(xi * xi, axis=2)
    yi2 = paddle.sum(yi * yi, axis=2)
    out = paddle.nn.functional.relu(yi2 * alpha - xi2 + margin)

    return paddle.mean(out)



class nTVLoss(nn.Layer):
    def __init__(self, weight=1):
        super(nTVLoss, self).__init__()
        self.weight = weight

    def forward(self, x, label):
        batch_size = x.shape[0]
        h_x = x.shape[2]
        w_x = x.shape[3]
        count_h = self._tensor_size(x[:, :, 1:, :])
        count_w = self._tensor_size(x[:, :, :, 1:])
        h_tv = paddle.pow((x[:, :, 1:, :] - x[:, :, :h_x - 1, :]), 2).sum()
        w_tv = paddle.pow((x[:, :, :, 1:] - x[:, :, :, :w_x - 1]), 2).sum()

        l_tv = paddle.pow((x[:, :, :h_x - 1, :w_x - 1] - x[:, :, 1:, 1:]), 2).sum()
        r_tv = paddle.pow((x[:, :, :h_x - 1, 1:w_x] - x[:, :, 1:, :w_x - 1]), 2).sum()

        return self.weight * 2 * (h_tv + l_tv + r_tv + w_tv ) / count_w / batch_size

    def _tensor_size(self, t):
        return t.shape[1] * t.shape[2] * t.shape[3]

class L1_Charbonnier_loss(nn.Layer):
    """L1 Charbonnierloss."""
    def __init__(self):
        super(L1_Charbonnier_loss, self).__init__()
        self.eps = 1e-6
 
    def forward(self, X, Y):
        diff = P.add(X, -Y)
        error = P.sqrt(diff * diff + self.eps)
        loss = P.mean(error)
        return loss

class MY_loss(nn.Layer):
    def __init__(self):
        super(MY_loss, self).__init__()
        self.l1closs = L1_Charbonnier_loss()
        self.mse = paddle.nn.MSELoss()
        self.vggex = LossVGG19()
        self.sobelloss = AdvSoberLoss()
 
    def forward(self, X, Y):
        loss1 = self.l1closs(X, Y)
        loss2 = self.mse(X, Y)

        NEW_X = paddle.concat([X,X,X],axis=1)
        NEW_Y = paddle.concat([Y,Y,Y],axis=1)

        loss4 = self.sobelloss(NEW_X, NEW_Y)

        NEW_X = self.vggex(NEW_X)
        NEW_Y = self.vggex(NEW_Y)

        loss3 = self.mse(NEW_X["relu2"],NEW_Y["relu2"])
        loss = loss1 + loss2 + 0.1 * loss3 + 0.1 * loss4
        return loss


if __name__ == '__main__':

    inp = P.randn((1, 1, 10, 10),dtype='float32')

    net = MY_loss()
    out = net(inp,inp)

    print('inp',inp.shape)
    print('out',out.shape)


# psnr_metric = PSNR()
# ssim_metric = SSIM()
# mse_metric = MSE()
# rmse_metric = RMSE()
# nrmse_metric = NRMSE()

# TV_criterion = nTVLoss()    # 用于规范图像噪声
# ssim_criterion = SSIMLoss() # 用于对原图进行
# sobel_loss = AdvSoberLoss()
# lossnet = LossVGG19() # 感知loss
# l1_loss = paddle.nn.L1Loss()
# Mse_loss = nn.MSELoss()

100%|██████████| 561203/561203 [00:07<00:00, 71149.32it/s]


inp [1, 1, 10, 10]
out [1]

9、自定义回调函数 ModelCheckpoint

class ModelCheckpoint(paddle.callbacks.Callback):
    """
    继承paddle.callbacks.Callback，该类的功能是
    训练模型时自动存储每轮训练得到的模型
    """
    def __init__(self, save_freq=1, save_dir=None):
        """
        构造函数实现
        """
        self.save_freq = save_freq
        self.save_dir = save_dir
   
    def on_epoch_begin(self, epoch=None, logs=None):
        """
        每轮训练开始前，获取当前轮数
        """
        self.epoch = epoch
   
    def _is_save(self):
        return self.model and self.save_dir and ParallelEnv().local_rank == 0
    
    # def on_epoch_end(self, epoch, logs=None):
    #     """
    #     每轮训练结束后，保存每轮的checkpoint
    #     """
    #     if self._is_save() and self.epoch % self.save_freq == 0:
    #         path = '{}/{}'.format(self.save_dir, epoch)
    #         print('save checkpoint at {}'.format(os.path.abspath(path)))
    #         self.model.save(path)
    
    def on_train_end(self, logs=None):
        """
        训练结束后，保存最后一轮的checkpoint
        """
        if self._is_save():
            path = '{}/final'.format(self.save_dir)
            print('save checkpoint at {}'.format(os.path.abspath(path)))
            self.model.save(path)

10、自定义模型

9.1、Sub_Pixel_CNN

class Sub_Pixel_CNN(nn.Layer):
# Sub_Pixel_CNN(upscale_factor=2, channels=1)
    def __init__(self, upscale_factor=2, channels=1):
        super(Sub_Pixel_CNN, self).__init__()
        
        self.conv1 = paddle.nn.Conv2D(channels,64,5,stride=1, padding=2)
        self.conv2 = paddle.nn.Conv2D(64,64,3,stride=1, padding=1)
        self.conv3 = paddle.nn.Conv2D(64,32,3,stride=1, padding=1)
        self.conv4 = paddle.nn.Conv2D(32,channels * (upscale_factor ** 2),3,stride=1, padding=1)

    def forward(self, x):
        x = self.conv1(x)
        x = self.conv2(x)
        x = self.conv3(x)
        x = self.conv4(x)
        x = paddle.nn.functional.pixel_shuffle(x,2)
        return x

9.2、VRCNN

# 定义模型结构
class VRCNN(nn.Layer):
    def __init__(self):
        super(VRCNN, self).__init__()
        
        self.conv1 = nn.Sequential(
            # 定义一个卷积层conv1，输入通道是1，输出通道64，卷积核大小为5，步长为1，padding为2，使用relu激活函数
            nn.Conv2D(1, 64, 5, stride=1, padding=2),
            nn.BatchNorm(64),
            nn.ReLU(),
        )
        self.conv2 = nn.Sequential(
            # 定义一个卷积层conv2，输入通道是64，输出通道16，卷积核大小为5，步长为1，padding为2，使用relu激活函数
            nn.Conv2D(64, 16, 5, stride=1, padding=2),
            nn.BatchNorm(16),
            nn.ReLU(),
        )
        self.conv3 = nn.Sequential(
            # 定义一个卷积层conv3，输入通道是64，输出通道32，卷积核大小为3，步长为1，padding为1，使用relu激活函数
            nn.Conv2D(64, 32, 3, stride=1, padding=1),
            nn.BatchNorm(32),
            nn.ReLU(),
        )
        self.conv4 = nn.Sequential(
            # 定义一个卷积层conv4，输入通道是48，输出通道16，卷积核大小为3，步长为1，padding为1，使用relu激活函数
            nn.Conv2D(48, 16, 3, stride=1, padding=1),
            nn.BatchNorm(16),
            nn.ReLU(),
        )
        self.conv5 = nn.Sequential(
            # 定义一个卷积层conv5，输入通道是48，输出通道32，卷积核大小为1，步长为1，padding为0，使用relu激活函数
            nn.Conv2D(48, 32, 1, stride=1, padding=0),
            nn.BatchNorm(32),
            nn.ReLU(),
        )
        self.conv6 = nn.Sequential(
            # 定义一个卷积层conv6，输入通道是48，输出通道1，卷积核大小为3，步长为1，padding为1，使用relu激活函数
            nn.Conv2D(48, 1, 3, stride=1, padding=1),
            nn.BatchNorm(1),
        )
        self.relu = nn.ReLU()

    # 定义网络的前向计算过程
    def forward(self, inputs):

        inputs = F.interpolate(x=inputs, scale_factor=[2,2], mode="bilinear")

        x = self.conv1(inputs)

        x1 = self.conv2(x)
        x2 = self.conv3(x)
        x = paddle.concat(x=[x1, x2], axis=1)

        x3 = self.conv4(x)
        x4 = self.conv5(x)
        x = paddle.concat(x=[x3, x4], axis=1)

        x = self.conv6(x)

        x = x + inputs
        x = self.relu(x)
        #print("x.shape = {}".format(x.shape))
        return x

if __name__ == '__main__':

    img_channel = 1
    width = 64

    net = VRCNN()

    inp = P.randn((1, 1, 120, 120),dtype='float32')

    out = net(inp)

    print('inp',inp.shape)
    print('out',out.shape)

inp [1, 1, 120, 120]
out [1, 1, 240, 240]

9.3、EDSR

from paddle.nn import Layer
from paddle import nn
import math

n_feat = 64
kernel_size = 3

# 残差块 尺寸不变
class _Res_Block(nn.Layer):
    def __init__(self):
        super(_Res_Block, self).__init__()
        self.res_conv = nn.Conv2D(n_feat, n_feat, kernel_size, padding=1)
        self.relu = nn.ReLU()

    def forward(self, x):
        y = self.relu(self.res_conv(x))
        y = self.res_conv(y)
        y *= 0.1
        # 残差加入
        y = paddle.add(y, x)
        return y


class EDSR(nn.Layer):
    def __init__(self):
        super(EDSR, self).__init__()

        in_ch = 1
        num_blocks = 32

        self.conv1 = nn.Conv2D(in_ch, n_feat, kernel_size, padding=1)
        # 扩大
        self.conv_up = nn.Conv2D(n_feat, n_feat * 4, kernel_size, padding=1)
        self.conv_out = nn.Conv2D(n_feat, in_ch, kernel_size, padding=1)

        self.body = self.make_layer(_Res_Block, num_blocks)
        # 上采样
        self.upsample = nn.Sequential(self.conv_up, nn.PixelShuffle(2))

    # 32个残差块
    def make_layer(self, block, layers):
        res_block = []
        for _ in range(layers):
            res_block.append(block())
        return nn.Sequential(*res_block)

    def forward(self, x):

        out = self.conv1(x)
        out = self.body(out)
        out = self.upsample(out)
        out = self.conv_out(out)

        return out

9.4、SRResNet

from paddle.nn import Layer
from paddle import nn
import math

# 是可以保持w，h不变的
class ConvolutionalBlock(nn.Layer):
    """
    卷积模块,由卷积层, BN归一化层, 激活层构成.
    """
    def __init__(self, in_channels, out_channels, kernel_size, stride=1, batch_norm=False, activation=None):
        """
        :参数 in_channels: 输入通道数
        :参数 out_channels: 输出通道数
        :参数 kernel_size: 核大小
        :参数 stride: 步长
        :参数 batch_norm: 是否包含BN层
        :参数 activation: 激活层类型; 如果没有则为None
        """
        super(ConvolutionalBlock, self).__init__()

        if activation is not None:
            activation = activation.lower()
            assert activation in {'prelu', 'leakyrelu', 'tanh'}

        # 层列表
        layers = list()

        # 1个卷积层
        layers.append(
            nn.Conv2D(in_channels=in_channels, out_channels=out_channels, kernel_size=kernel_size, stride=stride,
                      padding=kernel_size // 2))

        # 1个BN归一化层
        if batch_norm is True:
            layers.append(nn.BatchNorm2D(num_features=out_channels))

        # 1个激活层
        if activation == 'prelu':
            layers.append(nn.PReLU())
        elif activation == 'leakyrelu':
            layers.append(nn.LeakyReLU(0.2))
        elif activation == 'tanh':
            layers.append(nn.Tanh())

        # 合并层
        self.conv_block = nn.Sequential(*layers)

    def forward(self, input):
        output = self.conv_block(input)

        return output


# w,h放大
class SubPixelConvolutionalBlock(nn.Layer):

    def __init__(self, kernel_size=3, n_channels=64, scaling_factor=2):
        super(SubPixelConvolutionalBlock, self).__init__()

        # 首先通过卷积将通道数扩展为 scaling factor^2 倍
        self.conv = nn.Conv2D(in_channels=n_channels, out_channels=n_channels * (scaling_factor ** 2),
                              kernel_size=kernel_size, padding=kernel_size // 2)

        # 进行像素清洗，合并相关通道数据 放大了图像
        self.pixel_shuffle = nn.PixelShuffle(upscale_factor=scaling_factor)
        # 最后添加激活层
        self.prelu = nn.PReLU()

    def forward(self, input):

        output = self.conv(input)
        output = self.pixel_shuffle(output)  
        output = self.prelu(output) 

        return output


class ResidualBlock(nn.Layer):
    """
    残差模块, 包含两个卷积模块和一个跳连.
    """

    def __init__(self, kernel_size=3, n_channels=64):
        """
        :参数 kernel_size: 核大小
        :参数 n_channels: 输入和输出通道数（由于是ResNet网络，需要做跳连，因此输入和输出通道数是一致的）
        """
        super(ResidualBlock, self).__init__()

        # 第一个卷积块
        self.conv_block1 = ConvolutionalBlock(in_channels=n_channels, out_channels=n_channels, kernel_size=kernel_size,
                                              batch_norm=True, activation='PReLu')
        # 第二个卷积块
        self.conv_block2 = ConvolutionalBlock(in_channels=n_channels, out_channels=n_channels, kernel_size=kernel_size,
                                              batch_norm=True, activation=None)

    def forward(self, input):
        """
        前向传播.
        :参数 input: 输入图像集，张量表示，大小为 (N, n_channels, w, h)
        :返回: 输出图像集，张量表示，大小为 (N, n_channels, w, h)
        """
        residual = input  # (N, n_channels, w, h)
        output = self.conv_block1(input)  # (N, n_channels, w, h)
        output = self.conv_block2(output)  # (N, n_channels, w, h)
        output = output + residual  # (N, n_channels, w, h)

        return output


class SRResNet(nn.Layer):
# SRResNet(scaling_factor=2)
    def __init__(self, large_kernel_size=9, small_kernel_size=3, n_channels=64, n_blocks=16, scaling_factor=2):
        """
        :参数 large_kernel_size: 第一层卷积和最后一层卷积核大小
        :参数 small_kernel_size: 中间层卷积核大小
        :参数 n_channels: 中间层通道数
        :参数 n_blocks: 残差模块数
        :参数 scaling_factor: 放大比例
        """
        super(SRResNet, self).__init__()

        # 放大比例必须为 2、 4 或 8
        scaling_factor = int(scaling_factor)
        assert scaling_factor in {2, 4, 8}, "放大比例必须为 2、 4 或 8!"

        # 第一个卷积块
        self.conv_block1 = ConvolutionalBlock(in_channels=1, out_channels=n_channels, kernel_size=large_kernel_size,
                                              batch_norm=False, activation='PReLu')


        # 一系列残差模块, 每个残差模块包含一个跳连接
        self.residual_blocks = nn.Sequential(
            *[ResidualBlock(kernel_size=small_kernel_size, n_channels=n_channels) for i in range(n_blocks)])

        # 第二个卷积块
        self.conv_block2 = ConvolutionalBlock(in_channels=n_channels, out_channels=n_channels,
                                              kernel_size=small_kernel_size,
                                              batch_norm=True, activation=None)

        # 放大通过子像素卷积模块实现, 每个模块放大两倍 log2 2=1
        n_subpixel_convolution_blocks = int(math.log2(scaling_factor))
        self.subpixel_convolutional_blocks = nn.Sequential(
            *[SubPixelConvolutionalBlock(kernel_size=small_kernel_size, n_channels=n_channels, scaling_factor=2) for i
              in range(n_subpixel_convolution_blocks)])

        # 最后一个卷积模块
        self.conv_block3 = ConvolutionalBlock(in_channels=n_channels, out_channels=1, kernel_size=large_kernel_size,
                                              batch_norm=False, activation='Tanh')

    def forward(self, lr_imgs):
        """
        :参数 lr_imgs: 低分辨率输入图像集, 张量表示，大小为 (N, 3, w, h)
        :返回: 高分辨率输出图像集, 张量表示， 大小为 (N, 3, w * scaling factor, h * scaling factor)
        """
        output = self.conv_block1(lr_imgs)  # (16, 3, 24, 24)
        residual = output  # (16, 64, 24, 24)
        output = self.residual_blocks(output)  # (16, 64, 24, 24)
        output = self.conv_block2(output)  # (16, 64, 24, 24)
        output = output + residual  # (16, 64, 24, 24)
        output = self.subpixel_convolutional_blocks(output)  # (16, 64, 24 * 2, 24 * 2)
        sr_imgs = self.conv_block3(output)  # (16, 3, 24 * 2, 24 * 2)

        return sr_imgs

if __name__ == '__main__':

    img_channel = 1
    width = 64

    net = SRResNet()

    inp = P.randn((1, 1, 120, 120),dtype='float32')

    out = net(inp)

    print('inp',inp.shape)
    print('out',out.shape)

inp [1, 1, 120, 120]
out [1, 1, 240, 240]


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/norm.py:654: UserWarning: When training, we now always track global mean and variance.
  "When training, we now always track global mean and variance.")

9.5、修改的NAF模型

# cite： https://aistudio.baidu.com/aistudio/projectdetail/2884850?channelType=0&channel=0

import paddle
from paddle.fluid.layers.nn import transpose
import paddle.nn as nn
import math
import paddle.nn.functional as F


class simam_module(nn.Layer):
    def __init__(self, channels, e_lambda = 1e-4):
        super(simam_module, self).__init__()

        self.activaton = nn.Sigmoid()
        self.e_lambda = e_lambda

    def __repr__(self):
        s = self.__class__.__name__ + '('
        s += ('lambda=%f)' % self.e_lambda)
        return s

    @staticmethod
    def get_module_name():
        return "simam"

    def forward(self, x):

        b, c, h, w = x.shape
        
        n = w * h - 1

        x_minus_mu_square = (x - x.mean(axis=[2,3], keepdim=True)).pow(2)
        y = x_minus_mu_square / (4 * (x_minus_mu_square.sum(axis=[2,3], keepdim=True) / n + self.e_lambda)) + 0.5

        return x * self.activaton(y)
        # return x * y

if __name__ == '__main__':
    x = paddle.randn(shape=[1, 16, 64, 64])    # b, c, h, w

    simam = simam_module(16)
    y = simam(x)
    print(y.shape)

[1, 16, 64, 64]

class LayerNormFunction(P.autograd.PyLayer):

    @staticmethod
    def forward(ctx, x, weight, bias, eps):
        ctx.eps = eps
        N, C, H, W = x.shape[0],x.shape[1],x.shape[2],x.shape[3]
        mu = x.mean(1, keepdim=True)
        var = (x - mu).pow(2).mean(1, keepdim=True)
        y = (x - mu) / (var + eps).sqrt()
        ctx.save_for_backward(y, var, weight)
        y = weight.reshape([1, C, 1, 1]) * y + bias.reshape([1, C, 1, 1])
        return y

    @staticmethod
    def backward(ctx, grad_output):
        eps = ctx.eps

        N, C, H, W = grad_output.shape[0],grad_output.shape[1],grad_output.shape[2],grad_output.shape[3]
        # y, var, weight = ctx.saved_variables
        y, var, weight = ctx.saved_tensor()
        g = grad_output * weight.reshape([1, C, 1, 1])
        mean_g = g.mean(axis=1, keepdim=True)

        mean_gy = (g * y).mean(axis=1, keepdim=True)
        gx = 1. / P.sqrt(var + eps) * (g - y * mean_gy - mean_g)
        return gx, (grad_output * y).sum(axis=3).sum(axis=2).sum(axis=0), grad_output.sum(axis=3).sum(axis=2).sum(
            axis=0)

class LayerNorm2d(nn.Layer):

    def __init__(self, channels, eps=1e-6):
        super(LayerNorm2d, self).__init__()
        weight = P.static.create_parameter([channels], dtype='float32')
        bias = P.static.create_parameter([channels], dtype='float32')
        self.add_parameter('weight',weight)
        self.add_parameter('bias',bias)
        self.eps = eps

    def forward(self, x):
        return LayerNormFunction.apply(x, self.weight, self.bias, self.eps)

mylayer = LayerNorm2d(2)
for name, param in mylayer.named_parameters():
    print(name, param)      # will print w_tmp,_linear.weight,_linear.bias

weight Parameter containing:
Tensor(shape=[2], dtype=float32, place=Place(gpu:0), stop_gradient=False,
       [-0.01494403,  0.02245975])
bias Parameter containing:
Tensor(shape=[2], dtype=float32, place=Place(gpu:0), stop_gradient=False,
       [0.00556551, 0.00474699])

class SimpleGate(nn.Layer):
    def __init__(self):
        super(SimpleGate, self).__init__()

    def forward(self, x):
        x1, x2 = x.chunk(2, axis=1)
        # 在通道上拆分x为两个数据块
        return x1 * x2

class NAFBlock(nn.Layer):
    def __init__(self, c, DW_Expand=2, FFN_Expand=2, drop_out_rate=0.):
        super().__init__()
        dw_channel = c * DW_Expand
        self.conv1 = nn.Conv2D(in_channels=c, out_channels=dw_channel, kernel_size=1, padding=0, stride=1, groups=1)

        self.conv2 = nn.Conv2D(in_channels=dw_channel, out_channels=dw_channel, kernel_size=3, padding=1, stride=1,
                               groups=dw_channel)

        self.conv3 = nn.Conv2D(in_channels=dw_channel // 2, out_channels=c, kernel_size=1, padding=0, stride=1,
                               groups=1)

        # Simplified Channel Attention
        # self.sca = nn.Sequential(
        #     nn.AdaptiveAvgPool2D(1),
        #     nn.Conv2D(in_channels=dw_channel // 2, out_channels=dw_channel // 2, kernel_size=1, padding=0, stride=1,
        #               groups=1),
        # )

        self.sca = simam_module(dw_channel // 2)

        # SimpleGate
        self.sg = SimpleGate()

        ffn_channel = FFN_Expand * c
        self.conv4 = nn.Conv2D(in_channels=c, out_channels=ffn_channel, kernel_size=1, padding=0, stride=1, groups=1)
        self.conv5 = nn.Conv2D(in_channels=ffn_channel // 2, out_channels=c, kernel_size=1, padding=0, stride=1, groups=1)

        self.norm1 = LayerNorm2d(c)
        self.norm2 = LayerNorm2d(c)

        self.dropout1 = nn.Dropout(drop_out_rate) if drop_out_rate > 0. else nn.Identity()
        self.dropout2 = nn.Dropout(drop_out_rate) if drop_out_rate > 0. else nn.Identity()


        self.beta = P.create_parameter(shape=[1, c, 1, 1], dtype='float32')
        self.gamma = P.create_parameter(shape=[1, c, 1, 1], dtype='float32')

    def forward(self, inp):
        x = inp

        x = self.norm1(x)

        x = self.conv1(x)
        
        x = self.conv2(x)

        x = self.sg(x)

        x = self.sca(x)

        x = self.conv3(x)

        x = self.dropout1(x)

        y = inp + x * self.beta

        x = self.conv4(self.norm2(y))

        x = self.sg(x)

        x = self.sca(x)

        x = self.conv5(x)

        x = self.dropout2(x)

        return y + x * self.gamma


class NAFNet(nn.Layer):

    def __init__(self, img_channel=1, width=64,num_classes=1):
        super().__init__()
        self.num_classes = num_classes
        self.intro = nn.Conv2D(in_channels=img_channel, out_channels=width, kernel_size=3, padding=1, stride=1,
                               groups=1)
        self.ending = nn.Conv2D(in_channels=width//2, out_channels=img_channel, kernel_size=3, padding=1, stride=1,
                                groups=1)
        # self.ending = nn.Conv2D(in_channels=width, out_channels=img_channel, kernel_size=3, padding=1, stride=1,
        #                         groups=1)

        self.encoder_1 = nn.Sequential(NAFBlock(width),NAFBlock(width))
        self.encoder_2 = nn.Sequential(NAFBlock(width*2),NAFBlock(width*2))
        self.encoder_3 = nn.Sequential(NAFBlock(width*4),NAFBlock(width*4))
        self.encoder_4 = nn.Sequential(NAFBlock(width*8),NAFBlock(width*8))

        self.down_1 = nn.Conv2D(width, width*2, 2, 2)
        self.down_2 = nn.Conv2D(width*2, width*4, 2, 2)
        self.down_3 = nn.Conv2D(width*4, width*8, 2, 2)
        self.down_4 = nn.Conv2D(width*8, width*16, 2, 2)

        self.middle_blks = nn.Sequential(NAFBlock(width*16),NAFBlock(width*16))

        self.decoder_1 = nn.Sequential(NAFBlock(width),NAFBlock(width))
        self.decoder_2 = nn.Sequential(NAFBlock(width*2),NAFBlock(width*2))
        self.decoder_3 = nn.Sequential(NAFBlock(width*4),NAFBlock(width*4))
        self.decoder_4 = nn.Sequential(NAFBlock(width*8),NAFBlock(width*8))

        self.up_1 = nn.Sequential(nn.Conv2D(width*16, width*32, 1, bias_attr=False),nn.PixelShuffle(2))
        self.up_2 = nn.Sequential(nn.Conv2D(width*8, width*16, 1, bias_attr=False),nn.PixelShuffle(2))
        self.up_3 = nn.Sequential(nn.Conv2D(width*4, width*8, 1, bias_attr=False),nn.PixelShuffle(2))
        self.up_4 = nn.Sequential(nn.Conv2D(width*2, width*4, 1, bias_attr=False),nn.PixelShuffle(2))

        self.decoder_0 = nn.Sequential(NAFBlock(width//2),NAFBlock(width//2))
        self.up_5 = nn.Sequential(nn.Conv2D(width, width*2, 1, bias_attr=False),nn.PixelShuffle(2)) 

        self.padder_size = 2 ** 4
        

    def forward(self, inp):

        B, C, H, W = inp.shape

        inp = self.check_image_size(inp)
        
        x = self.intro(inp)

        encs = []

        # -------------------------------编码层
        x = self.encoder_1(x)
        encs.append(x)
        
        x = self.down_1(x)

        x = self.encoder_2(x)
        encs.append(x)

        x = self.down_2(x)

        x = self.encoder_3(x)
        encs.append(x)

        x = self.down_3(x)

        x = self.encoder_4(x)
        encs.append(x)

        x = self.down_4(x)

        # -------------------------------中间层

        x = self.middle_blks(x)

        # -------------------------------解码层

        x = self.up_1(x)
        x = x + encs[::-1][0] # 倒叙排列找第一个（找倒数第一个）
        x = self.decoder_4(x)

        x = self.up_2(x)
        x = x + encs[::-1][1]
        x = self.decoder_3(x)

        x = self.up_3(x)
        x = x + encs[::-1][2]
        x = self.decoder_2(x)

        x = self.up_4(x)
        x = x + encs[::-1][3]
        x = self.decoder_1(x)

        ori_x = F.interpolate(x=inp, scale_factor=[2,2], mode="bilinear")
        x = self.up_5(x)
        x = self.decoder_0(x)


        # -------------------------------维度变化层
        x = self.ending(x)
        x = x + ori_x

        return x[:, :, :H*SCALE, :W*SCALE]

    def check_image_size(self, x):
        h, w = x.shape[2],x.shape[3]
        mod_pad_h = (self.padder_size - h % self.padder_size) % self.padder_size
        mod_pad_w = (self.padder_size - w % self.padder_size) % self.padder_size
        x = F.pad(x, (0, mod_pad_w, 0, mod_pad_h))
        return x



if __name__ == '__main__':

    SCALE = 2
    img_channel = 1
    width = 64

    net = NAFNet(img_channel=img_channel, width=width)

    inp = P.randn((1, 1, 120, 120),dtype='float32')

    out = net(inp)

    print('inp',inp.shape)
    print('out',out.shape)

inp [1, 1, 120, 120]
out [1, 1, 240, 240]

11、模型训练

train_dataset= MyDataset(mode='train')
valid_dataset= MyDataset(mode='valid')
test_dataset = MyDataset(mode='test')

model1 = paddle.Model(Sub_Pixel_CNN(upscale_factor=SCALE, channels=1))
model2 = paddle.Model(VRCNN())
model3 = paddle.Model(EDSR())
model4 = paddle.Model(SRResNet(scaling_factor=SCALE))
model5 = paddle.Model(NAFNet(img_channel=1, width=32))

Model_name = 'Sub_Pixel_CNN_KT2'
model = model1

#一旦不再使用即释放内存垃圾，=1.0 垃圾占用内存大小达到10G时，释放内存垃圾
%set_env FLAGS_eager_delete_tensor_gb=0.0
#启用快速垃圾回收策略，不等待cuda kernel 结束，直接释放显存
!export FLAGS_fast_eager_deletion_mode=1
#该环境变量设置只占用0%的显存
!export FLAGS_fraction_of_gpu_memory_to_use=0

paddle.device.cuda.empty_cache()

env: FLAGS_eager_delete_tensor_gb=0.0

# https://www.paddlepaddle.org.cn/documentation/docs/zh/practices/cv/super_resolution_sub_pixel.html#sub-pixel
# L1_Charbonnier_loss


# 设置学习率
base_lr = 0.001
# lr = paddle.optimizer.lr.PolynomialDecay(base_lr, power=0.9, end_lr=0.0001,decay_steps=2000)
# lr = paddle.optimizer.lr.CosineAnnealingDecay(learning_rate=base_lr, T_max=10000*5, verbose=False)

# optimizer = paddle.optimizer.Momentum(lr, parameters=model.parameters(), momentum=0.9, weight_decay=4.0e-5)

optimizer = paddle.optimizer.Adam(learning_rate=base_lr,parameters=model.parameters())

model.prepare(optimizer,
              MY_loss(),
            #   L1_Charbonnier_loss(),
              [PSNR(),SSIM(),MSE(),RMSE(),NRMSE()],
             )

# 启动模型训练，指定训练数据集，设置训练轮次，设置每次数据集计算的批次大小，设置日志格式

model.fit(train_dataset,
          epochs=EPOCH,
          batch_size=64,
          verbose=1,
          callbacks=ModelCheckpoint(save_dir= Model_name+'_checkpoint'))

The loss value printed in the log is the current step, and the metric is the average value of previous steps.
Epoch 1/20
step 750/750 [==============================] - loss: 0.0285 - psnr: 32.2469 - ssim: 0.9469 - mse: 6.5497e-04 - rmse: 0.0255 - nrmse: 0.1244 - 802ms/step          
Epoch 2/20
step 750/750 [==============================] - loss: 0.0281 - psnr: 32.3239 - ssim: 0.9482 - mse: 6.3479e-04 - rmse: 0.0252 - nrmse: 0.1227 - 804ms/step          
Epoch 3/20
step 750/750 [==============================] - loss: 0.0276 - psnr: 32.3412 - ssim: 0.9483 - mse: 6.3187e-04 - rmse: 0.0251 - nrmse: 0.1225 - 804ms/step          
Epoch 4/20
step 120/750 [===>..........................] - loss: 0.0277 - psnr: 32.2940 - ssim: 0.9481 - mse: 6.4101e-04 - rmse: 0.0253 - nrmse: 0.1234 - ETA: 8:27 - 806ms/step


---------------------------------------------------------------------------

KeyboardInterrupt                         Traceback (most recent call last)

/tmp/ipykernel_8623/4189507737.py in <module>
     24           batch_size=64,
     25           verbose=1,
---> 26           callbacks=ModelCheckpoint(save_dir= Model_name+'_checkpoint'))


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in fit(self, train_data, eval_data, batch_size, epochs, eval_freq, log_freq, save_dir, save_freq, verbose, drop_last, shuffle, num_workers, callbacks, accumulate_grad_batches, num_iters)
   1765         for epoch in range(epochs):
   1766             cbks.on_epoch_begin(epoch)
-> 1767             logs = self._run_one_epoch(train_loader, cbks, 'train')
   1768             cbks.on_epoch_end(epoch, logs)
   1769 


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in _run_one_epoch(self, data_loader, callbacks, mode, logs)
   2095                                    step + 1 == len(data_loader))
   2096 
-> 2097                 outs = getattr(self, mode + '_batch')(*_inputs)
   2098 
   2099                 if self._metrics and self._loss:


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in train_batch(self, inputs, labels, update)
   1091               print(loss)
   1092         """
-> 1093         loss = self._adapter.train_batch(inputs, labels, update)
   1094         if fluid._non_static_mode() and self._input_info is None:
   1095             self._update_inputs()


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in train_batch(self, inputs, labels, update)
    747         for metric in self.model._metrics:
    748             metric_outs = metric.compute(*(to_list(outputs) + labels))
--> 749             m = metric.update(*[to_numpy(m) for m in to_list(metric_outs)])
    750             metrics.append(m)
    751 


/tmp/ipykernel_8623/1449683858.py in update(self, preds, labels)
     87         for b in range(B):
     88             for c in range(C):
---> 89                 temp += skimage.metrics.structural_similarity(preds[b,c,:,:], labels[b,c,:,:], data_range=1)
     90 
     91         temp = temp/(B*C)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/skimage/_shared/utils.py in fixed_func(*args, **kwargs)
    346 
    347             # Call the function with the fixed arguments
--> 348             return func(*args, **kwargs)
    349 
    350         if func.__doc__ is not None:


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/skimage/metrics/_structural_similarity.py in structural_similarity(im1, im2, win_size, gradient, data_range, channel_axis, multichannel, gaussian_weights, full, **kwargs)
    214     uxx = filter_func(im1 * im1, **filter_args)
    215     uyy = filter_func(im2 * im2, **filter_args)
--> 216     uxy = filter_func(im1 * im2, **filter_args)
    217     vx = cov_norm * (uxx - ux * ux)
    218     vy = cov_norm * (uyy - uy * uy)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/ndimage/filters.py in uniform_filter(input, size, output, mode, cval, origin)
    961         for axis, size, origin, mode in axes:
    962             uniform_filter1d(input, int(size), axis, output, mode,
--> 963                              cval, origin)
    964             input = output
    965     else:


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/ndimage/filters.py in uniform_filter1d(input, size, axis, output, mode, cval, origin)
    896     if not complex_output:
    897         _nd_image.uniform_filter1d(input, size, axis, output, mode, cval,
--> 898                                    origin)
    899     else:
    900         _nd_image.uniform_filter1d(input.real, size, axis, output.real, mode,


KeyboardInterrupt: 

Model_name = 'VRCNN_KT2'
model = model2

#一旦不再使用即释放内存垃圾，=1.0 垃圾占用内存大小达到10G时，释放内存垃圾
!export FLAGS_eager_delete_tensor_gb=0.0
#启用快速垃圾回收策略，不等待cuda kernel 结束，直接释放显存
!export FLAGS_fast_eager_deletion_mode=1
#该环境变量设置只占用0%的显存
!export FLAGS_fraction_of_gpu_memory_to_use=0

paddle.device.cuda.empty_cache()

# https://www.paddlepaddle.org.cn/documentation/docs/zh/practices/cv/super_resolution_sub_pixel.html#sub-pixel
# L1_Charbonnier_loss
model.prepare(paddle.optimizer.Adam(learning_rate=0.001,parameters=model.parameters()),
              MY_loss(),
              [PSNR(),SSIM(),MSE(),RMSE(),NRMSE()],
             )

# 启动模型训练，指定训练数据集，设置训练轮次，设置每次数据集计算的批次大小，设置日志格式

model.fit(train_dataset,
          epochs=EPOCH,
          batch_size=64,
          verbose=1,
          callbacks=ModelCheckpoint(save_dir= Model_name+'_checkpoint'))

The loss value printed in the log is the current step, and the metric is the average value of previous steps.
Epoch 1/20
step  10/750 [..............................] - loss: 0.1422 - psnr: 28.5244 - ssim: 0.9194 - mse: 0.0014 - rmse: 0.0375 - nrmse: 0.9546 - ETA: 1:33 - 127ms/step

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/skimage/metrics/simple_metrics.py:108: RuntimeWarning: divide by zero encountered in double_scalars
  return np.sqrt(mean_squared_error(image_true, image_test)) / denom


step 740/750 [============================>.] - loss: 0.0487 - psnr: 33.7226 - ssim: 0.9830 - mse: 4.2436e-04 - rmse: 0.0206 - nrmse: 0.1310 - ETA: 1s - 114ms/step

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/skimage/metrics/simple_metrics.py:108: RuntimeWarning: invalid value encountered in double_scalars
  return np.sqrt(mean_squared_error(image_true, image_test)) / denom


step 750/750 [==============================] - loss: 0.0371 - psnr: 34.6075 - ssim: 0.9854 - mse: 3.4614e-04 - rmse: 0.0186 - nrmse: 0.1098 - 114ms/step          
Epoch 2/20
step 750/750 [==============================] - loss: 0.0489 - psnr: 33.1561 - ssim: 0.9842 - mse: 4.8349e-04 - rmse: 0.0220 - nrmse: 0.1089 - 114ms/step          
Epoch 3/20
step 750/750 [==============================] - loss: 0.0351 - psnr: 35.4124 - ssim: 0.9825 - mse: 2.8758e-04 - rmse: 0.0170 - nrmse: 0.1623 - 114ms/step          
Epoch 4/20
step 750/750 [==============================] - loss: 0.0473 - psnr: 33.7377 - ssim: 0.9848 - mse: 4.2289e-04 - rmse: 0.0206 - nrmse: 0.0904 - 114ms/step          
Epoch 5/20
step 750/750 [==============================] - loss: 0.0437 - psnr: 34.1153 - ssim: 0.9840 - mse: 3.8767e-04 - rmse: 0.0197 - nrmse: 0.1561 - 114ms/step          
Epoch 6/20
step 750/750 [==============================] - loss: 0.0386 - psnr: 33.9412 - ssim: 0.9834 - mse: 4.0354e-04 - rmse: 0.0201 - nrmse: 0.1215 - 114ms/step          
Epoch 7/20
step 750/750 [==============================] - loss: 0.0368 - psnr: 34.5516 - ssim: 0.9792 - mse: 3.5062e-04 - rmse: 0.0187 - nrmse: 0.1966 - 114ms/step          
Epoch 8/20
step 750/750 [==============================] - loss: 0.0415 - psnr: 34.1747 - ssim: 0.9864 - mse: 3.8241e-04 - rmse: 0.0196 - nrmse: 0.1426 - 114ms/step          
Epoch 9/20
step 750/750 [==============================] - loss: 0.0414 - psnr: 33.8262 - ssim: 0.9850 - mse: 4.1436e-04 - rmse: 0.0204 - nrmse: 0.1098 - 114ms/step          
Epoch 10/20
step 750/750 [==============================] - loss: 0.0417 - psnr: 34.0265 - ssim: 0.9840 - mse: 3.9569e-04 - rmse: 0.0199 - nrmse: 0.1023 - 114ms/step          
Epoch 11/20
step 750/750 [==============================] - loss: 0.0412 - psnr: 34.7837 - ssim: 0.9793 - mse: 3.3238e-04 - rmse: 0.0182 - nrmse: 0.1182 - 115ms/step          
Epoch 12/20
step 750/750 [==============================] - loss: 0.0412 - psnr: 34.8368 - ssim: 0.9841 - mse: 3.2833e-04 - rmse: 0.0181 - nrmse: 0.1122 - 116ms/step          
Epoch 13/20
step 750/750 [==============================] - loss: 0.0438 - psnr: 34.1078 - ssim: 0.9803 - mse: 3.8835e-04 - rmse: 0.0197 - nrmse: 0.1176 - 116ms/step          
Epoch 14/20
step 750/750 [==============================] - loss: 0.0361 - psnr: 34.1508 - ssim: 0.9870 - mse: 3.8452e-04 - rmse: 0.0196 - nrmse: 0.1081 - 115ms/step          
Epoch 15/20
step 750/750 [==============================] - loss: 0.0354 - psnr: 34.8307 - ssim: 0.9860 - mse: 3.2880e-04 - rmse: 0.0181 - nrmse: 0.1224 - 115ms/step          
Epoch 16/20
step 750/750 [==============================] - loss: 0.0349 - psnr: 34.4778 - ssim: 0.9843 - mse: 3.5663e-04 - rmse: 0.0189 - nrmse: 0.1706 - 115ms/step          
Epoch 17/20
step 750/750 [==============================] - loss: 0.0388 - psnr: 34.1937 - ssim: 0.9861 - mse: 3.8074e-04 - rmse: 0.0195 - nrmse: 0.1076 - 115ms/step          
Epoch 18/20
step 750/750 [==============================] - loss: 0.0422 - psnr: 34.2011 - ssim: 0.9864 - mse: 3.8009e-04 - rmse: 0.0195 - nrmse: 0.1138 - 115ms/step          
Epoch 19/20
step 750/750 [==============================] - loss: 0.0366 - psnr: 34.6230 - ssim: 0.9832 - mse: 3.4490e-04 - rmse: 0.0186 - nrmse: 0.1062 - 115ms/step          
Epoch 20/20
step 750/750 [==============================] - loss: 0.0403 - psnr: 33.4524 - ssim: 0.9858 - mse: 4.5160e-04 - rmse: 0.0213 - nrmse: 0.0937 - 115ms/step          
save checkpoint at /home/aistudio/VRCNN_checkpoint/final

Model_name = 'EDSR'
model = model3

#一旦不再使用即释放内存垃圾，=1.0 垃圾占用内存大小达到10G时，释放内存垃圾
!export FLAGS_eager_delete_tensor_gb=0.0
#启用快速垃圾回收策略，不等待cuda kernel 结束，直接释放显存
!export FLAGS_fast_eager_deletion_mode=1
#该环境变量设置只占用0%的显存
!export FLAGS_fraction_of_gpu_memory_to_use=0

paddle.device.cuda.empty_cache()

# https://www.paddlepaddle.org.cn/documentation/docs/zh/practices/cv/super_resolution_sub_pixel.html#sub-pixel
# L1_Charbonnier_loss

model.prepare(paddle.optimizer.Adam(learning_rate=0.001,parameters=model.parameters()),
              MY_loss(),
              [PSNR(),SSIM(),MSE(),RMSE(),NRMSE()],
             )

# 启动模型训练，指定训练数据集，设置训练轮次，设置每次数据集计算的批次大小，设置日志格式

model.fit(train_dataset,
          epochs=EPOCH,
          batch_size=64,
          verbose=1,
          callbacks=ModelCheckpoint(save_dir= Model_name+'_checkpoint'))

The loss value printed in the log is the current step, and the metric is the average value of previous steps.
Epoch 1/20
step 750/750 [==============================] - loss: 0.0358 - psnr: 35.0711 - ssim: 0.9850 - mse: 3.1109e-04 - rmse: 0.0176 - nrmse: 0.1273 - 162ms/step          
Epoch 2/20
step 750/750 [==============================] - loss: 0.0389 - psnr: 34.6819 - ssim: 0.9742 - mse: 3.4026e-04 - rmse: 0.0184 - nrmse: 0.1203 - 161ms/step          
Epoch 3/20
step 750/750 [==============================] - loss: 0.0353 - psnr: 34.9515 - ssim: 0.9858 - mse: 3.1978e-04 - rmse: 0.0179 - nrmse: 0.1314 - 161ms/step          
Epoch 4/20
step 750/750 [==============================] - loss: 0.0387 - psnr: 34.0316 - ssim: 0.9832 - mse: 3.9523e-04 - rmse: 0.0199 - nrmse: 0.1263 - 161ms/step          
Epoch 5/20
step 750/750 [==============================] - loss: 0.0326 - psnr: 35.0452 - ssim: 0.9825 - mse: 3.1295e-04 - rmse: 0.0177 - nrmse: 0.1232 - 162ms/step          
Epoch 6/20
step 750/750 [==============================] - loss: 0.0381 - psnr: 33.9176 - ssim: 0.9772 - mse: 4.0573e-04 - rmse: 0.0201 - nrmse: 0.1302 - 161ms/step          
Epoch 7/20
step 750/750 [==============================] - loss: 0.0309 - psnr: 34.8910 - ssim: 0.9874 - mse: 3.2426e-04 - rmse: 0.0180 - nrmse: 0.1021 - 162ms/step          
Epoch 8/20
step 750/750 [==============================] - loss: 0.0269 - psnr: 35.3550 - ssim: 0.9731 - mse: 2.9140e-04 - rmse: 0.0171 - nrmse: 0.1649 - 161ms/step          
Epoch 9/20
step 750/750 [==============================] - loss: 0.0277 - psnr: 35.8470 - ssim: 0.9868 - mse: 2.6019e-04 - rmse: 0.0161 - nrmse: 0.1101 - 162ms/step          
Epoch 10/20
step 750/750 [==============================] - loss: 0.0317 - psnr: 35.2562 - ssim: 0.9849 - mse: 2.9811e-04 - rmse: 0.0173 - nrmse: 0.1011 - 161ms/step          
Epoch 11/20
step 750/750 [==============================] - loss: 0.0286 - psnr: 35.8863 - ssim: 0.9848 - mse: 2.5785e-04 - rmse: 0.0161 - nrmse: 0.1155 - 161ms/step          
Epoch 12/20
step 750/750 [==============================] - loss: 0.0270 - psnr: 35.1926 - ssim: 0.9780 - mse: 3.0251e-04 - rmse: 0.0174 - nrmse: 0.1363 - 161ms/step          
Epoch 13/20
step 750/750 [==============================] - loss: 0.0257 - psnr: 36.4744 - ssim: 0.9892 - mse: 2.2520e-04 - rmse: 0.0150 - nrmse: 0.1164 - 161ms/step          
Epoch 14/20
step 750/750 [==============================] - loss: 0.0349 - psnr: 33.4152 - ssim: 0.9856 - mse: 4.5549e-04 - rmse: 0.0213 - nrmse: 0.1081 - 162ms/step          
Epoch 15/20
step 750/750 [==============================] - loss: 0.0307 - psnr: 35.4264 - ssim: 0.9876 - mse: 2.8665e-04 - rmse: 0.0169 - nrmse: 0.0904 - 162ms/step          
Epoch 16/20
step 750/750 [==============================] - loss: 0.0337 - psnr: 33.5216 - ssim: 0.9710 - mse: 4.4447e-04 - rmse: 0.0211 - nrmse: 0.1380 - 161ms/step          
Epoch 17/20
step 750/750 [==============================] - loss: 0.0270 - psnr: 34.9437 - ssim: 0.9873 - mse: 3.2035e-04 - rmse: 0.0179 - nrmse: 0.1125 - 161ms/step          
Epoch 18/20
step 750/750 [==============================] - loss: 0.0337 - psnr: 35.3052 - ssim: 0.9850 - mse: 2.9477e-04 - rmse: 0.0172 - nrmse: 0.0970 - 161ms/step          
Epoch 19/20
step 750/750 [==============================] - loss: 0.0276 - psnr: 35.3846 - ssim: 0.9861 - mse: 2.8943e-04 - rmse: 0.0170 - nrmse: 0.1080 - 161ms/step          
Epoch 20/20
step 750/750 [==============================] - loss: 0.0289 - psnr: 34.7869 - ssim: 0.9869 - mse: 3.3213e-04 - rmse: 0.0182 - nrmse: 0.0866 - 161ms/step          
save checkpoint at /home/aistudio/EDSR_checkpoint/final

# 无框架测试代码
from tqdm import tqdm_notebook

train_loader = paddle.io.DataLoader(train_dataset, batch_size=64, shuffle=True)

model = Sub_Pixel_CNN(upscale_factor=SCALE, channels=1)

metric1,metric2 = PSNR(),SSIM()

model.train()

# 设置迭代次数
epochs = 5

# 设置优化器
# scheduler = paddle.optimizer.lr.StepDecay(learning_rate=0.01, step_size=2, gamma=0.8, verbose=True)
# optim = paddle.optimizer.SGD(learning_rate=scheduler, parameters=model.parameters())
optim = paddle.optimizer.Adam(learning_rate=0.001,parameters=model.parameters())

# 设置损失函数
loss_fn = MY_loss()
for epoch in range(epochs):

    psnr_list = []
    ssim_list = []
    
    for data in tqdm_notebook(train_loader()):
        
        x_data = data[0]            # 训练数据
        y_data = data[1]            # 训练数据标签
        predicts = model(x_data)    # 预测结果  

        psnr = metric1.update(predicts,y_data)
        ssim = metric2.update(predicts,y_data)

        psnr_list.append(psnr)
        ssim_list.append(ssim)
        
        # 计算损失 等价于 prepare 中loss的设置
        loss = loss_fn(predicts, y_data)
        
        # print("epoch: {}, loss is: {},PSNR is: {},SSIM is: {}".format(epoch, loss.numpy(),psnr,ssim))

        loss.backward()

        # 更新参数 
        optim.step()
        # 梯度清零
        optim.clear_grad()

    print("epoch: {}, loss is: {},PSNR is: {},SSIM is: {}".format(epoch, loss.numpy(),np.array(psnr_list).mean(),np.array(ssim_list).mean()))
    scheduler.step()

HBox(children=(IntProgress(value=0, max=750), HTML(value='')))


epoch: 0, loss is: [0.06278042],PSNR is: 30.16339282961605,SSIM is: 0.9426153815529351
Epoch 1: StepDecay set learning rate to 0.01.



HBox(children=(IntProgress(value=0, max=750), HTML(value='')))


epoch: 1, loss is: [0.06349419],PSNR is: 32.24472592343079,SSIM is: 0.97570674496347
Epoch 2: StepDecay set learning rate to 0.008.



HBox(children=(IntProgress(value=0, max=750), HTML(value='')))


epoch: 2, loss is: [0.06702027],PSNR is: 32.394738155912236,SSIM is: 0.9733830994603897
Epoch 3: StepDecay set learning rate to 0.008.



HBox(children=(IntProgress(value=0, max=750), HTML(value='')))


epoch: 3, loss is: [0.07004125],PSNR is: 32.341874329713,SSIM is: 0.9739344468748179
Epoch 4: StepDecay set learning rate to 0.006400000000000001.



HBox(children=(IntProgress(value=0, max=750), HTML(value='')))


epoch: 4, loss is: [0.07502048],PSNR is: 32.517667913098045,SSIM is: 0.9707958158608372
Epoch 5: StepDecay set learning rate to 0.006400000000000001.

Model_name = 'SRResNet'
model = model4

#一旦不再使用即释放内存垃圾，=1.0 垃圾占用内存大小达到10G时，释放内存垃圾
!export FLAGS_eager_delete_tensor_gb=0.0
#启用快速垃圾回收策略，不等待cuda kernel 结束，直接释放显存
!export FLAGS_fast_eager_deletion_mode=1
#该环境变量设置只占用0%的显存
!export FLAGS_fraction_of_gpu_memory_to_use=0

paddle.device.cuda.empty_cache()

# https://www.paddlepaddle.org.cn/documentation/docs/zh/practices/cv/super_resolution_sub_pixel.html#sub-pixel
# L1_Charbonnier_loss
model.prepare(paddle.optimizer.Adam(learning_rate=0.001,parameters=model.parameters()),
              MY_loss(),
              [PSNR(),SSIM(),MSE(),RMSE(),NRMSE()],
             )

# 启动模型训练，指定训练数据集，设置训练轮次，设置每次数据集计算的批次大小，设置日志格式

model.fit(train_dataset,
          epochs=EPOCH,
          batch_size=64,
          verbose=1,
          callbacks=ModelCheckpoint(save_dir= Model_name+'_checkpoint'))

The loss value printed in the log is the current step, and the metric is the average value of previous steps.
Epoch 1/20


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/norm.py:654: UserWarning: When training, we now always track global mean and variance.
  "When training, we now always track global mean and variance.")


step  40/750 [>.............................] - loss: 0.1193 - psnr: 30.2991 - ssim: 0.8928 - mse: 0.0023 - rmse: 0.0482 - nrmse: 0.4371 - ETA: 1:33 - 132ms/step


---------------------------------------------------------------------------

KeyboardInterrupt                         Traceback (most recent call last)

/tmp/ipykernel_171/3920091531.py in <module>
     12           batch_size=64,
     13           verbose=1,
---> 14           callbacks=ModelCheckpoint(save_dir= Model_name+'_checkpoint'))


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in fit(self, train_data, eval_data, batch_size, epochs, eval_freq, log_freq, save_dir, save_freq, verbose, drop_last, shuffle, num_workers, callbacks, accumulate_grad_batches, num_iters)
   1765         for epoch in range(epochs):
   1766             cbks.on_epoch_begin(epoch)
-> 1767             logs = self._run_one_epoch(train_loader, cbks, 'train')
   1768             cbks.on_epoch_end(epoch, logs)
   1769 


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in _run_one_epoch(self, data_loader, callbacks, mode, logs)
   2095                                    step + 1 == len(data_loader))
   2096 
-> 2097                 outs = getattr(self, mode + '_batch')(*_inputs)
   2098 
   2099                 if self._metrics and self._loss:


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in train_batch(self, inputs, labels, update)
   1091               print(loss)
   1092         """
-> 1093         loss = self._adapter.train_batch(inputs, labels, update)
   1094         if fluid._non_static_mode() and self._input_info is None:
   1095             self._update_inputs()


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in train_batch(self, inputs, labels, update)
    726             else:
    727                 outputs = self.model.network.forward(
--> 728                     *[to_variable(x) for x in inputs])
    729 
    730         losses = self.model._loss(*(to_list(outputs) + labels))


/tmp/ipykernel_171/3325794139.py in forward(self, lr_imgs)
    156         output = self.conv_block1(lr_imgs)  # (16, 3, 24, 24)
    157         residual = output  # (16, 64, 24, 24)
--> 158         output = self.residual_blocks(output)  # (16, 64, 24, 24)
    159         output = self.conv_block2(output)  # (16, 64, 24, 24)
    160         output = output + residual  # (16, 64, 24, 24)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in __call__(self, *inputs, **kwargs)
    928             return self.forward(*inputs, **kwargs)
    929         else:
--> 930             return self._dygraph_call_func(*inputs, **kwargs)
    931 
    932     def forward(self, *inputs, **kwargs):


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in _dygraph_call_func(self, *inputs, **kwargs)
    913                 outputs = self.forward(*inputs, **kwargs)
    914         else:
--> 915             outputs = self.forward(*inputs, **kwargs)
    916 
    917         for forward_post_hook in self._forward_post_hooks.values():


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/container.py in forward(self, input)
     96     def forward(self, input):
     97         for layer in self._sub_layers.values():
---> 98             input = layer(input)
     99         return input
    100 


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in __call__(self, *inputs, **kwargs)
    928             return self.forward(*inputs, **kwargs)
    929         else:
--> 930             return self._dygraph_call_func(*inputs, **kwargs)
    931 
    932     def forward(self, *inputs, **kwargs):


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in _dygraph_call_func(self, *inputs, **kwargs)
    913                 outputs = self.forward(*inputs, **kwargs)
    914         else:
--> 915             outputs = self.forward(*inputs, **kwargs)
    916 
    917         for forward_post_hook in self._forward_post_hooks.values():


/tmp/ipykernel_171/3325794139.py in forward(self, input)
    102         """
    103         residual = input  # (N, n_channels, w, h)
--> 104         output = self.conv_block1(input)  # (N, n_channels, w, h)
    105         output = self.conv_block2(output)  # (N, n_channels, w, h)
    106         output = output + residual  # (N, n_channels, w, h)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in __call__(self, *inputs, **kwargs)
    928             return self.forward(*inputs, **kwargs)
    929         else:
--> 930             return self._dygraph_call_func(*inputs, **kwargs)
    931 
    932     def forward(self, *inputs, **kwargs):


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in _dygraph_call_func(self, *inputs, **kwargs)
    913                 outputs = self.forward(*inputs, **kwargs)
    914         else:
--> 915             outputs = self.forward(*inputs, **kwargs)
    916 
    917         for forward_post_hook in self._forward_post_hooks.values():


/tmp/ipykernel_171/3325794139.py in forward(self, input)
     47 
     48     def forward(self, input):
---> 49         output = self.conv_block(input)
     50 
     51         return output


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in __call__(self, *inputs, **kwargs)
    928             return self.forward(*inputs, **kwargs)
    929         else:
--> 930             return self._dygraph_call_func(*inputs, **kwargs)
    931 
    932     def forward(self, *inputs, **kwargs):


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in _dygraph_call_func(self, *inputs, **kwargs)
    913                 outputs = self.forward(*inputs, **kwargs)
    914         else:
--> 915             outputs = self.forward(*inputs, **kwargs)
    916 
    917         for forward_post_hook in self._forward_post_hooks.values():


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/container.py in forward(self, input)
     96     def forward(self, input):
     97         for layer in self._sub_layers.values():
---> 98             input = layer(input)
     99         return input
    100 


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in __call__(self, *inputs, **kwargs)
    928             return self.forward(*inputs, **kwargs)
    929         else:
--> 930             return self._dygraph_call_func(*inputs, **kwargs)
    931 
    932     def forward(self, *inputs, **kwargs):


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/layers.py in _dygraph_call_func(self, *inputs, **kwargs)
    913                 outputs = self.forward(*inputs, **kwargs)
    914         else:
--> 915             outputs = self.forward(*inputs, **kwargs)
    916 
    917         for forward_post_hook in self._forward_post_hooks.values():


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/layer/norm.py in forward(self, input)
    664             epsilon=self._epsilon,
    665             data_format=self._data_format,
--> 666             use_global_stats=self._use_global_stats)
    667 
    668     def extra_repr(self):


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/nn/functional/norm.py in batch_norm(x, running_mean, running_var, weight, bias, training, momentum, epsilon, data_format, use_global_stats, name)
    204             batch_norm_out, _, _, _, _, _ = _C_ops.batch_norm(
    205                 x, weight, bias, running_mean, running_var, None, mean_out,
--> 206                 variance_out, *attrs)
    207 
    208         return dygraph_utils._append_activation_in_dygraph(


KeyboardInterrupt: 

Model_name = 'NAFNet'
model = model5

#一旦不再使用即释放内存垃圾，=1.0 垃圾占用内存大小达到10G时，释放内存垃圾
!export FLAGS_eager_delete_tensor_gb=0.0
#启用快速垃圾回收策略，不等待cuda kernel 结束，直接释放显存
!export FLAGS_fast_eager_deletion_mode=1
#该环境变量设置只占用0%的显存
!export FLAGS_fraction_of_gpu_memory_to_use=0

paddle.device.cuda.empty_cache()

# https://www.paddlepaddle.org.cn/documentation/docs/zh/practices/cv/super_resolution_sub_pixel.html#sub-pixel
# L1_Charbonnier_loss

model.prepare(paddle.optimizer.Adam(learning_rate=0.001,parameters=model.parameters()),
              MY_loss(),
              [PSNR(),SSIM(),MSE(),RMSE(),NRMSE()],
             )

# 启动模型训练，指定训练数据集，设置训练轮次，设置每次数据集计算的批次大小，设置日志格式

model.fit(train_dataset,
          epochs=EPOCH,
          batch_size=64,
          verbose=1,
          callbacks=ModelCheckpoint(save_dir= Model_name+'_checkpoint'))

The loss value printed in the log is the current step, and the metric is the average value of previous steps.
Epoch 1/20
step 750/750 [==============================] - loss: 0.0425 - psnr: 34.5434 - ssim: 0.9839 - mse: 3.5128e-04 - rmse: 0.0187 - nrmse: 0.1335 - 294ms/step          
Epoch 2/20
step 750/750 [==============================] - loss: 0.0445 - psnr: 34.1515 - ssim: 0.9844 - mse: 3.8446e-04 - rmse: 0.0196 - nrmse: 0.1245 - 293ms/step          
Epoch 3/20
step 750/750 [==============================] - loss: 0.0434 - psnr: 34.1930 - ssim: 0.9838 - mse: 3.8080e-04 - rmse: 0.0195 - nrmse: 0.1098 - 296ms/step          
Epoch 4/20
step 750/750 [==============================] - loss: 0.0407 - psnr: 34.1296 - ssim: 0.9841 - mse: 3.8640e-04 - rmse: 0.0197 - nrmse: 0.1101 - 295ms/step          
Epoch 5/20
step 750/750 [==============================] - loss: 0.0312 - psnr: 35.6564 - ssim: 0.9873 - mse: 2.7187e-04 - rmse: 0.0165 - nrmse: 0.1174 - 295ms/step          
Epoch 6/20
step 750/750 [==============================] - loss: 0.0376 - psnr: 34.4083 - ssim: 0.9852 - mse: 3.6239e-04 - rmse: 0.0190 - nrmse: 0.1244 - 301ms/step          
Epoch 7/20
step 750/750 [==============================] - loss: 0.0368 - psnr: 34.9323 - ssim: 0.9859 - mse: 3.2119e-04 - rmse: 0.0179 - nrmse: 0.1401 - 299ms/step          
Epoch 8/20
step 750/750 [==============================] - loss: 0.0341 - psnr: 34.4772 - ssim: 0.9859 - mse: 3.5668e-04 - rmse: 0.0189 - nrmse: 0.1346 - 298ms/step          
Epoch 9/20
step 750/750 [==============================] - loss: 0.0296 - psnr: 35.2572 - ssim: 0.9865 - mse: 2.9805e-04 - rmse: 0.0173 - nrmse: 0.1641 - 300ms/step          
Epoch 10/20
step 750/750 [==============================] - loss: 0.0394 - psnr: 33.7732 - ssim: 0.9862 - mse: 4.1945e-04 - rmse: 0.0205 - nrmse: 0.0894 - 297ms/step          
Epoch 11/20
step 750/750 [==============================] - loss: 0.0376 - psnr: 34.7627 - ssim: 0.9855 - mse: 3.3399e-04 - rmse: 0.0183 - nrmse: 0.1177 - 299ms/step          
Epoch 12/20
step 750/750 [==============================] - loss: 0.0395 - psnr: 34.0623 - ssim: 0.9843 - mse: 3.9244e-04 - rmse: 0.0198 - nrmse: 0.1184 - 295ms/step          
Epoch 13/20
step 750/750 [==============================] - loss: 0.0399 - psnr: 34.1988 - ssim: 0.9849 - mse: 3.8029e-04 - rmse: 0.0195 - nrmse: 0.0977 - 291ms/step          
Epoch 14/20
step 750/750 [==============================] - loss: 0.0409 - psnr: 33.7524 - ssim: 0.9840 - mse: 4.2146e-04 - rmse: 0.0205 - nrmse: 0.1161 - 291ms/step          
Epoch 15/20
step 750/750 [==============================] - loss: 0.0343 - psnr: 34.9239 - ssim: 0.9866 - mse: 3.2182e-04 - rmse: 0.0179 - nrmse: 0.1187 - 290ms/step          
Epoch 16/20
step 750/750 [==============================] - loss: 0.0367 - psnr: 33.5507 - ssim: 0.9860 - mse: 4.4150e-04 - rmse: 0.0210 - nrmse: 0.1378 - 291ms/step          
Epoch 17/20
step 750/750 [==============================] - loss: 0.0376 - psnr: 34.2966 - ssim: 0.9872 - mse: 3.7182e-04 - rmse: 0.0193 - nrmse: 0.1026 - 291ms/step          
Epoch 18/20
step 750/750 [==============================] - loss: 0.0360 - psnr: 34.7681 - ssim: 0.9863 - mse: 3.3357e-04 - rmse: 0.0183 - nrmse: 0.1251 - 291ms/step          
Epoch 19/20
step 750/750 [==============================] - loss: 0.0304 - psnr: 35.6507 - ssim: 0.9880 - mse: 2.7223e-04 - rmse: 0.0165 - nrmse: 0.0863 - 290ms/step          
Epoch 20/20
step 750/750 [==============================] - loss: 0.0355 - psnr: 33.9343 - ssim: 0.9868 - mse: 4.0418e-04 - rmse: 0.0201 - nrmse: 0.1094 - 289ms/step          
save checkpoint at /home/aistudio/NAFNet_checkpoint/final

12、模型评估

eval_result = model.evaluate(valid_dataset, verbose=1)
print(eval_result)

# Sub_Pixel_CNN + MSELOSS
# {'loss': [0.00049995194], 'psnr': 33.01071776396916, 'ssim': 0.8393247210309024, 'mse': 0.0004999519027504247, 'rmse': 0.022359604261936852, 'nrmse': 0.07611498313403414}
# Sub_Pixel_CNN + L1_Charbonnier_loss
# {'loss': [0.008971893], 'psnr': 35.19330590838505, 'ssim': 0.9752850247905843, 'mse': 0.0003024610209004461, 'rmse': 0.017391406524500715, 'nrmse': 0.0604310206128466}

# NAF
# {'loss': [0.011660097], 'psnr': 32.93446294296621, 'ssim': 0.9871351872573194, 'mse': 0.0005088077371462047, 'rmse': 0.022556766992328593, 'nrmse': 0.07637058937234419}

# {'loss': [0.038608607], 'psnr': 34.248901708670175, 'ssim': 0.9831008942602794, 'mse': 0.00037593246552586524, 'rmse': 0.0193889779391763, 'nrmse': 0.06675509143967433}


# 2x
# Sub_Pixel_CNN

# SRResNet
# {'loss': [0.03413957], 'psnr': 32.70351634926379, 'ssim': 0.988856885854074, 'mse': 0.0005365971631781785, 'rmse': 0.02316456697584003, 'nrmse': 0.07811984548710951}

Eval begin...
step 576/576 [==============================] - loss: 0.0188 - psnr: 31.8475 - ssim: 0.9442 - mse: 6.9162e-04 - rmse: 0.0260 - nrmse: 0.1239 - 13ms/step          
Eval samples: 576
{'loss': [0.01876361], 'psnr': 31.84752944566835, 'ssim': 0.9442151274630379, 'mse': 0.000691616901588113, 'rmse': 0.026032056683165192, 'nrmse': 0.12387659958605875}

eval_result = model.evaluate(test_dataset, verbose=1)
print(eval_result)

# Sub_Pixel_CNN + MSELOSS
# {'loss': [0.00052807305], 'psnr': 32.773060184192445, 'ssim': 0.8255382422525163, 'mse': 0.000528073017899273, 'rmse': 0.02297983937931841, 'nrmse': 0.08957864419897979}
# Sub_Pixel_CNN + L1_Charbonnier_loss
# {'loss': [0.011634254], 'psnr': 33.146820283049486, 'ssim': 0.9711140393078723, 'mse': 0.00048452698250778573, 'rmse': 0.02201197361682468, 'nrmse': 0.08558840001112705}

# {'loss': [0.047101058], 'psnr': 32.48981346615865, 'ssim': 0.9734736951423864, 'mse': 0.0005636618529631433, 'rmse': 0.023741563827244894, 'nrmse': 0.09272385819962056}

# SRResNet+100
# {'loss': [0.037792806], 'psnr': 33.4587757793354, 'ssim': 0.9772896100415559, 'mse': 0.00045094379861308314, 'rmse': 0.021235437330393812, 'nrmse': 0.0821697023088623}


# 2x
# Sub_Pixel_CNN

# SRResNet
# {'loss': [0.037792806], 'psnr': 33.4587757793354, 'ssim': 0.9772896100415559, 'mse': 0.00045094379861308314, 'rmse': 0.021235437330393812, 'nrmse': 0.0821697023088623}

Eval begin...
step 6720/6720 [==============================] - loss: 0.0122 - psnr: 32.4111 - ssim: 0.9482 - mse: 6.3074e-04 - rmse: 0.0245 - nrmse: 0.1189 - 13ms/step          
Eval samples: 6720
{'loss': [0.012182929], 'psnr': 32.411140988755356, 'ssim': 0.9482212071483026, 'mse': 0.0006307391969291598, 'rmse': 0.024511901592441154, 'nrmse': 0.11893688483562116}

13、模型加载

model1 = paddle.Model(Sub_Pixel_CNN(upscale_factor=SCALE, channels=1))
model2 = paddle.Model(VRCNN())
model3 = paddle.Model(EDSR())
model4 = paddle.Model(SRResNet(scaling_factor=SCALE))
model5 = paddle.Model(NAFNet(img_channel=1, width=32))


path1 = './'+'Sub_Pixel_CNN'+'_checkpoint/final'
path2 = './'+'VRCNN'+'_checkpoint/final'
path3 = './'+'EDSR'+'_checkpoint/final'
path4 = './'+'SRResNet'+'_checkpoint/final'
path5 = './'+'NAFNet'+'_checkpoint/final'

model1.load(path1)
model2.load(path2)
model3.load(path3)
model4.load(path4)
model5.load(path5)

model1.prepare(paddle.optimizer.Adam(learning_rate=0.001,parameters=model.parameters()),
              MY_loss(),
              [PSNR(),SSIM(),MSE(),RMSE(),NRMSE()],
             )
model2.prepare(paddle.optimizer.Adam(learning_rate=0.001,parameters=model.parameters()),
              MY_loss(),
              [PSNR(),SSIM(),MSE(),RMSE(),NRMSE()],
             )
model3.prepare(paddle.optimizer.Adam(learning_rate=0.001,parameters=model.parameters()),
              MY_loss(),
              [PSNR(),SSIM(),MSE(),RMSE(),NRMSE()],
             )
model4.prepare(paddle.optimizer.Adam(learning_rate=0.001,parameters=model.parameters()),
              MY_loss(),
              [PSNR(),SSIM(),MSE(),RMSE(),NRMSE()],
             )
model5.prepare(paddle.optimizer.Adam(learning_rate=0.001,parameters=model.parameters()),
              MY_loss(),
              [PSNR(),SSIM(),MSE(),RMSE(),NRMSE()],
             )

eval_result = model1.evaluate(valid_dataset, verbose=1)
print(eval_result)
eval_result = model2.evaluate(valid_dataset, verbose=1)
print(eval_result)
eval_result = model3.evaluate(valid_dataset, verbose=1)
print(eval_result)
eval_result = model4.evaluate(valid_dataset, verbose=1)
print(eval_result)
eval_result = model5.evaluate(valid_dataset, verbose=1)
print(eval_result)

Eval begin...
step 576/576 [==============================] - loss: 0.0418 - psnr: 34.6720 - ssim: 0.9837 - mse: 3.4104e-04 - rmse: 0.0185 - nrmse: 0.0641 - 15ms/step          
Eval samples: 576
{'loss': [0.0418299], 'psnr': 34.671996326352456, 'ssim': 0.9837169891626211, 'mse': 0.00034103611113178863, 'rmse': 0.01846716305044683, 'nrmse': 0.06409965785369509}
Eval begin...
step 230/576 [==========>...................] - loss: 0.1022 - psnr: 26.9783 - ssim: 0.9560 - mse: 0.0020 - rmse: 0.0448 - nrmse: 0.1148 - ETA: 5s - 17ms/step    


---------------------------------------------------------------------------

KeyboardInterrupt                         Traceback (most recent call last)

/tmp/ipykernel_171/3964492600.py in <module>
      1 eval_result = model1.evaluate(valid_dataset, verbose=1)
      2 print(eval_result)
----> 3 eval_result = model2.evaluate(valid_dataset, verbose=1)
      4 print(eval_result)
      5 eval_result = model3.evaluate(valid_dataset, verbose=1)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in evaluate(self, eval_data, batch_size, log_freq, verbose, num_workers, callbacks, num_iters)
   1876                        'metrics': self._metrics_name()})
   1877 
-> 1878         logs = self._run_one_epoch(eval_loader, cbks, 'eval')
   1879 
   1880         cbks.on_end('eval', logs)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in _run_one_epoch(self, data_loader, callbacks, mode, logs)
   2095                                    step + 1 == len(data_loader))
   2096 
-> 2097                 outs = getattr(self, mode + '_batch')(*_inputs)
   2098 
   2099                 if self._metrics and self._loss:


<decorator-gen-539> in eval_batch(self, inputs, labels)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/dygraph/base.py in _decorate_function(func, *args, **kwargs)
    352         def _decorate_function(func, *args, **kwargs):
    353             with self:
--> 354                 return func(*args, **kwargs)
    355 
    356         @decorator.decorator


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in eval_batch(self, inputs, labels)
   1143               print(loss)
   1144         """
-> 1145         loss = self._adapter.eval_batch(inputs, labels)
   1146         if fluid._non_static_mode() and self._input_info is None:
   1147             self._update_inputs()


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/hapi/model.py in eval_batch(self, inputs, labels)
    792 
    793             metric_outs = metric.compute(*(to_list(outputs) + labels))
--> 794             m = metric.update(*[to_numpy(m) for m in to_list(metric_outs)])
    795             metrics.append(m)
    796 


/tmp/ipykernel_171/1428740584.py in update(self, preds, labels)
     81         for b in range(B):
     82             for c in range(C):
---> 83                 temp += skimage.metrics.structural_similarity(preds[b,c,:,:], labels[b,c,:,:], data_range=1)
     84 
     85         self.ssim = temp/(B*C)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/skimage/_shared/utils.py in fixed_func(*args, **kwargs)
    346 
    347             # Call the function with the fixed arguments
--> 348             return func(*args, **kwargs)
    349 
    350         if func.__doc__ is not None:


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/skimage/metrics/_structural_similarity.py in structural_similarity(im1, im2, win_size, gradient, data_range, channel_axis, multichannel, gaussian_weights, full, **kwargs)
    212 
    213     # compute (weighted) variances and covariances
--> 214     uxx = filter_func(im1 * im1, **filter_args)
    215     uyy = filter_func(im2 * im2, **filter_args)
    216     uxy = filter_func(im1 * im2, **filter_args)


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/ndimage/filters.py in uniform_filter(input, size, output, mode, cval, origin)
    961         for axis, size, origin, mode in axes:
    962             uniform_filter1d(input, int(size), axis, output, mode,
--> 963                              cval, origin)
    964             input = output
    965     else:


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/scipy/ndimage/filters.py in uniform_filter1d(input, size, axis, output, mode, cval, origin)
    896     if not complex_output:
    897         _nd_image.uniform_filter1d(input, size, axis, output, mode, cval,
--> 898                                    origin)
    899     else:
    900         _nd_image.uniform_filter1d(input.real, size, axis, output.real, mode,


KeyboardInterrupt: 

eval_result = model1.evaluate(test_dataset, verbose=1)
print(eval_result)
eval_result = model2.evaluate(test_dataset, verbose=1)
print(eval_result)
eval_result = model3.evaluate(test_dataset, verbose=1)
print(eval_result)
eval_result = model4.evaluate(test_dataset, verbose=1)
print(eval_result)
eval_result = model5.evaluate(test_dataset, verbose=1)
print(eval_result)

Eval begin...
step 6720/6720 [==============================] - loss: 0.0451 - psnr: 31.4026 - ssim: 0.9732 - mse: 7.2400e-04 - rmse: 0.0269 - nrmse: 0.1068 - 15ms/step          
Eval samples: 6720
{'loss': [0.045064367], 'psnr': 31.40263306685403, 'ssim': 0.9731983550192798, 'mse': 0.0007239968777774099, 'rmse': 0.02690719007584051, 'nrmse': 0.10683957374720605}
Eval begin...
step 6720/6720 [==============================] - loss: 0.0472 - psnr: 32.0030 - ssim: 0.9732 - mse: 6.3052e-04 - rmse: 0.0251 - nrmse: 0.0988 - 16ms/step          
Eval samples: 6720
{'loss': [0.04722685], 'psnr': 32.003038932843296, 'ssim': 0.9732293535427508, 'mse': 0.000630515992729399, 'rmse': 0.025110077513408813, 'nrmse': 0.0987667310072532}
Eval begin...
step 6720/6720 [==============================] - loss: 0.0443 - psnr: 30.4647 - ssim: 0.9732 - mse: 8.9852e-04 - rmse: 0.0300 - nrmse: 0.1207 - 29ms/step          
Eval samples: 6720
{'loss': [0.044340976], 'psnr': 30.4647164184756, 'ssim': 0.9731898243521414, 'mse': 0.0008985212614511, 'rmse': 0.029975344225731587, 'nrmse': 0.1207084483046749}
Eval begin...
step 6720/6720 [==============================] - loss: 0.0443 - psnr: 30.3831 - ssim: 0.9682 - mse: 9.1556e-04 - rmse: 0.0303 - nrmse: 0.1220 - 29ms/step          
Eval samples: 6720
{'loss': [0.044347625], 'psnr': 30.38311330263867, 'ssim': 0.9682021174133281, 'mse': 0.0009155639196116994, 'rmse': 0.030258286792409436, 'nrmse': 0.12200117075121371}
Eval begin...
step 6720/6720 [==============================] - loss: 0.0462 - psnr: 31.8775 - ssim: 0.9696 - mse: 6.4900e-04 - rmse: 0.0255 - nrmse: 0.1003 - 106ms/step          
Eval samples: 6720
{'loss': [0.046169166], 'psnr': 31.87754186468502, 'ssim': 0.9695507521587402, 'mse': 0.0006490016688202694, 'rmse': 0.025475511159155754, 'nrmse': 0.10031246245285091}

14、模型推断

# 查看MRI图像
import matplotlib.pyplot as plt
import random

model = model5

# predict_results = model.predict(test_dataset)[0]
predict_results = model.predict(valid_dataset)[0]
# 从测试集中取出一张图片

index = random.randint(0,len(predict_results))

# img, label = test_dataset[index]
img, label = valid_dataset[index]

predict = predict_results[index]
fig = plt.figure(figsize=(10,10))
plt.subplot(1,3,1)
plt.imshow(img[0],cmap='gray')
plt.subplot(1,3,2)
plt.imshow(predict[0][0],cmap='gray')
plt.subplot(1,3,3)
plt.imshow(label[0],cmap='gray')

plt.show()

Predict begin...
step 576/576 [==============================] - 74ms/step          
Predict samples: 576


/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/image.py:425: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  a_min = np.asscalar(a_min.astype(scaled_dtype))
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/image.py:426: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  a_max = np.asscalar(a_max.astype(scaled_dtype))

五、总结

• 本项目分析了多个超分辨率模型对MR图像超分的重建结果，我们发现模型很容易过拟合，由于MR图像颜色结构过于单一，即使ssim已经达到很高但psnr却无法接近40，提高训练次数也无法解决。针对此现象，需要提出一种新的结构或方法进一步解决此问题。

参考文献

[1] WANG Z， LIU D， YANG J， et al. Deep networks for image Super-Resolution with sparse prior[C]//2015 IEEE International Conference on Computer Vision （ICCV）， Santiago， 2016： 370-378.

[2] WEI W， FENG G， ZHANG D， et al. Accurate single image super-resolution using cascading dense connections[J]. Electronics Letters， 2015， 55（13）： 739-742.

[3]HONG X， ZAN Y， WANG W， et al. Enhancing the image quality via transferred deep residual learning of coarse PET sinograms[J]. IEEE Transactions on Medical Imaging， 2018， 37（10）：2322-2332.

[4] DONG C， LOY C C， TANG X. Accelerating the Super-Resolution convolutional neural network[C]// 14th European Conference on Computer Vision. Amsterdam： Springer International Publishing. Amsterdam， 2016： 391-407.

[5] Zhao X , Zhang Y , Zhang T , et al. Channel Splitting Network for Single MR Image Super-Resolution[J]. IEEE Transactions on Image Processing, 2019, 28(99):5649-5662.

[6] http://brain-development.org/ixi-dataset/

此文仅为搬运，原作链接：https://aistudio.baidu.com/aistudio/projectdetail/4314871?channelType=0&channel=0