基于PaddlePaddle 新工具 API 的【猴痘识别】
本文通过猴痘识别，介绍数据集新API DatasetFolder 和 random_split 使用，并通过HAPI和自定义CNN模型进行了全过程实现，有助于初学者学习掌握PaddlePaddle。
·
一、基于PaddlePaddle 新工具 API 的【猴痘识别】
- 还记否，曾经开始学习神经网络，需要手动遍历文件夹，手动生成数据列表？
- 还记否，曾经划分数据集，不断for循环抽取数据？
多看看飞桨 API， 以前的问题现在都可以轻松得到解决。
本文通过猴痘识别例子，介绍 数据集常用工具 DatasetFolder 和 random_split 使用。
本文采取了2种方式进行模型训练预测：
- 加载PaddlePaddle内置模型库并使用 HAPI 进行训练预测
- 自定义CNN模型并使用普通API进行训练预测
本教程可以较好的加速PaddlePaddle学习理解。
二、数据集介绍
- 数据集文件名为45-data.zip。
- 该数据集包含2个类别不同病症图像。
- 图像大小不一，格式为.jpg。
1.数据解压缩
# 数据解压缩
!unzip -qoa data/data172723/45-data.zip -d 45-data
2.导入必要python包
# 导入需要的包
import shutil
import tempfile
import cv2
import numpy as np
import paddle
import paddle.vision.transforms as T
from pathlib import Path
from paddle.vision.datasets import DatasetFolder
import os,PIL,pathlib
print(paddle.version.cuda())
print(paddle.__version__)
11.2
2.3.2
3.使用DatasetFolder读取数据集
此处只需一句 DatasetFolder 类即可，而以往需要写个get_data_list。。。。。。
total_datadir = './45-data/'
train_transforms = T.Compose([
T.Resize([224, 224]), # 将输入图片resize成统一尺寸
T.RandomRotation(degrees=(-10, 10)), # 随机旋转，-10到10度之间随机选
T.RandomHorizontalFlip(), # 随机水平翻转 选择一个概率概率
T.RandomVerticalFlip(), # 随机垂直翻转
T.ToTensor(), # 将PIL Image或numpy.ndarray转换为tensor，并归一化到[0,1]之间
T.Normalize( # 标准化处理-->转换为标准正太分布（高斯分布），使模型更容易收敛
mean=[0.485, 0.456, 0.406],
std = [0.229, 0.224, 0.225]) # 其中 mean=[0.485,0.456,0.406]与std=[0.229,0.224,0.225] 从数据集中随机抽样计算得到的。
])
total_data = DatasetFolder(total_datadir,transform=train_transforms)
# 打印数据长度
print(len(total_data))
# 打印第一个数据，Tensor格式的 img 、label
print(total_data[0])
2142
(Tensor(shape=[3, 224, 224], dtype=float32, place=Place(gpu:0), stop_gradient=True,
[[[-2.11790395, -2.11790395, -2.11790395, ..., -2.11790395,
-2.11790395, -2.11790395],
[-2.11790395, -2.11790395, -2.11790395, ..., -2.11790395,
-2.11790395, -2.11790395],
[-2.11790395, -2.11790395, -2.11790395, ..., -2.11790395,
-2.11790395, -2.11790395],
...,
[-2.11790395, -2.11790395, -2.11790395, ..., -2.11790395,
-2.11790395, -2.11790395],
[-2.11790395, -2.11790395, -2.11790395, ..., -2.11790395,
-2.11790395, -2.11790395],
[-2.11790395, -2.11790395, -2.11790395, ..., -2.11790395,
-2.11790395, -2.11790395]],
[[-2.03571415, -2.03571415, -2.03571415, ..., -2.03571415,
-2.03571415, -2.03571415],
[-2.03571415, -2.03571415, -2.03571415, ..., -2.03571415,
-2.03571415, -2.03571415],
[-2.03571415, -2.03571415, -2.03571415, ..., -2.03571415,
-2.03571415, -2.03571415],
...,
[-2.03571415, -2.03571415, -2.03571415, ..., -2.03571415,
-2.03571415, -2.03571415],
[-2.03571415, -2.03571415, -2.03571415, ..., -2.03571415,
-2.03571415, -2.03571415],
[-2.03571415, -2.03571415, -2.03571415, ..., -2.03571415,
-2.03571415, -2.03571415]],
[[-1.80444443, -1.80444443, -1.80444443, ..., -1.80444443,
-1.80444443, -1.80444443],
[-1.80444443, -1.80444443, -1.80444443, ..., -1.80444443,
-1.80444443, -1.80444443],
[-1.80444443, -1.80444443, -1.80444443, ..., -1.80444443,
-1.80444443, -1.80444443],
...,
[-1.80444443, -1.80444443, -1.80444443, ..., -1.80444443,
-1.80444443, -1.80444443],
[-1.80444443, -1.80444443, -1.80444443, ..., -1.80444443,
-1.80444443, -1.80444443],
[-1.80444443, -1.80444443, -1.80444443, ..., -1.80444443,
-1.80444443, -1.80444443]]]), 0)
W1018 00:52:19.770149 1698 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 11.2, Runtime API Version: 11.2
W1018 00:52:19.773958 1698 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
4.random_split切分数据集
random_split(total_data, [train_size, test_size])，一下搞定，而以往还在for循环中。。。。。。
from paddle.io import random_split
train_size = int(0.8 * len(total_data))
test_size = len(total_data) - train_size
train_dataset, test_dataset = random_split(total_data, [train_size, test_size])
train_dataset, test_dataset
(<paddle.fluid.dataloader.dataset.Subset at 0x7f2dcde15e90>,
<paddle.fluid.dataloader.dataset.Subset at 0x7f2dcde15d90>)
print(len(train_dataset),len(test_dataset))
1713 429
三、使用现有模型库模型
使用自定义CNN模型可以跳过三，直接运行第四部分。
1.加载MobileNetV3Large模型
# 模型组网并初始化网络
mobilenetv3 = paddle.vision.models.MobileNetV3Large(num_classes=2)
# 可视化模型组网结构和参数
paddle.summary(mobilenetv3,(1, 3, 224, 224))
-------------------------------------------------------------------------------
Layer (type) Input Shape Output Shape Param #
===============================================================================
Conv2D-1 [[1, 3, 224, 224]] [1, 16, 112, 112] 432
BatchNorm2D-1 [[1, 16, 112, 112]] [1, 16, 112, 112] 64
Hardswish-1 [[1, 16, 112, 112]] [1, 16, 112, 112] 0
Conv2D-2 [[1, 16, 112, 112]] [1, 16, 112, 112] 144
BatchNorm2D-2 [[1, 16, 112, 112]] [1, 16, 112, 112] 64
ReLU-1 [[1, 16, 112, 112]] [1, 16, 112, 112] 0
Conv2D-3 [[1, 16, 112, 112]] [1, 16, 112, 112] 256
BatchNorm2D-3 [[1, 16, 112, 112]] [1, 16, 112, 112] 64
InvertedResidual-1 [[1, 16, 112, 112]] [1, 16, 112, 112] 0
Conv2D-4 [[1, 16, 112, 112]] [1, 64, 112, 112] 1,024
BatchNorm2D-4 [[1, 64, 112, 112]] [1, 64, 112, 112] 256
ReLU-2 [[1, 64, 112, 112]] [1, 64, 112, 112] 0
Conv2D-5 [[1, 64, 112, 112]] [1, 64, 56, 56] 576
BatchNorm2D-5 [[1, 64, 56, 56]] [1, 64, 56, 56] 256
ReLU-3 [[1, 64, 56, 56]] [1, 64, 56, 56] 0
Conv2D-6 [[1, 64, 56, 56]] [1, 24, 56, 56] 1,536
BatchNorm2D-6 [[1, 24, 56, 56]] [1, 24, 56, 56] 96
InvertedResidual-2 [[1, 16, 112, 112]] [1, 24, 56, 56] 0
Conv2D-7 [[1, 24, 56, 56]] [1, 72, 56, 56] 1,728
BatchNorm2D-7 [[1, 72, 56, 56]] [1, 72, 56, 56] 288
ReLU-4 [[1, 72, 56, 56]] [1, 72, 56, 56] 0
Conv2D-8 [[1, 72, 56, 56]] [1, 72, 56, 56] 648
BatchNorm2D-8 [[1, 72, 56, 56]] [1, 72, 56, 56] 288
ReLU-5 [[1, 72, 56, 56]] [1, 72, 56, 56] 0
Conv2D-9 [[1, 72, 56, 56]] [1, 24, 56, 56] 1,728
BatchNorm2D-9 [[1, 24, 56, 56]] [1, 24, 56, 56] 96
InvertedResidual-3 [[1, 24, 56, 56]] [1, 24, 56, 56] 0
Conv2D-10 [[1, 24, 56, 56]] [1, 72, 56, 56] 1,728
BatchNorm2D-10 [[1, 72, 56, 56]] [1, 72, 56, 56] 288
ReLU-6 [[1, 72, 56, 56]] [1, 72, 56, 56] 0
Conv2D-11 [[1, 72, 56, 56]] [1, 72, 28, 28] 1,800
BatchNorm2D-11 [[1, 72, 28, 28]] [1, 72, 28, 28] 288
ReLU-7 [[1, 72, 28, 28]] [1, 72, 28, 28] 0
AdaptiveAvgPool2D-1 [[1, 72, 28, 28]] [1, 72, 1, 1] 0
Conv2D-12 [[1, 72, 1, 1]] [1, 24, 1, 1] 1,752
ReLU-8 [[1, 24, 1, 1]] [1, 24, 1, 1] 0
Conv2D-13 [[1, 24, 1, 1]] [1, 72, 1, 1] 1,800
Hardsigmoid-1 [[1, 72, 1, 1]] [1, 72, 1, 1] 0
SqueezeExcitation-1 [[1, 72, 28, 28]] [1, 72, 28, 28] 0
Conv2D-14 [[1, 72, 28, 28]] [1, 40, 28, 28] 2,880
BatchNorm2D-12 [[1, 40, 28, 28]] [1, 40, 28, 28] 160
InvertedResidual-4 [[1, 24, 56, 56]] [1, 40, 28, 28] 0
Conv2D-15 [[1, 40, 28, 28]] [1, 120, 28, 28] 4,800
BatchNorm2D-13 [[1, 120, 28, 28]] [1, 120, 28, 28] 480
ReLU-9 [[1, 120, 28, 28]] [1, 120, 28, 28] 0
Conv2D-16 [[1, 120, 28, 28]] [1, 120, 28, 28] 3,000
BatchNorm2D-14 [[1, 120, 28, 28]] [1, 120, 28, 28] 480
ReLU-10 [[1, 120, 28, 28]] [1, 120, 28, 28] 0
AdaptiveAvgPool2D-2 [[1, 120, 28, 28]] [1, 120, 1, 1] 0
Conv2D-17 [[1, 120, 1, 1]] [1, 32, 1, 1] 3,872
ReLU-11 [[1, 32, 1, 1]] [1, 32, 1, 1] 0
Conv2D-18 [[1, 32, 1, 1]] [1, 120, 1, 1] 3,960
Hardsigmoid-2 [[1, 120, 1, 1]] [1, 120, 1, 1] 0
SqueezeExcitation-2 [[1, 120, 28, 28]] [1, 120, 28, 28] 0
Conv2D-19 [[1, 120, 28, 28]] [1, 40, 28, 28] 4,800
BatchNorm2D-15 [[1, 40, 28, 28]] [1, 40, 28, 28] 160
InvertedResidual-5 [[1, 40, 28, 28]] [1, 40, 28, 28] 0
Conv2D-20 [[1, 40, 28, 28]] [1, 120, 28, 28] 4,800
BatchNorm2D-16 [[1, 120, 28, 28]] [1, 120, 28, 28] 480
ReLU-12 [[1, 120, 28, 28]] [1, 120, 28, 28] 0
Conv2D-21 [[1, 120, 28, 28]] [1, 120, 28, 28] 3,000
BatchNorm2D-17 [[1, 120, 28, 28]] [1, 120, 28, 28] 480
ReLU-13 [[1, 120, 28, 28]] [1, 120, 28, 28] 0
AdaptiveAvgPool2D-3 [[1, 120, 28, 28]] [1, 120, 1, 1] 0
Conv2D-22 [[1, 120, 1, 1]] [1, 32, 1, 1] 3,872
ReLU-14 [[1, 32, 1, 1]] [1, 32, 1, 1] 0
Conv2D-23 [[1, 32, 1, 1]] [1, 120, 1, 1] 3,960
Hardsigmoid-3 [[1, 120, 1, 1]] [1, 120, 1, 1] 0
SqueezeExcitation-3 [[1, 120, 28, 28]] [1, 120, 28, 28] 0
Conv2D-24 [[1, 120, 28, 28]] [1, 40, 28, 28] 4,800
BatchNorm2D-18 [[1, 40, 28, 28]] [1, 40, 28, 28] 160
InvertedResidual-6 [[1, 40, 28, 28]] [1, 40, 28, 28] 0
Conv2D-25 [[1, 40, 28, 28]] [1, 240, 28, 28] 9,600
BatchNorm2D-19 [[1, 240, 28, 28]] [1, 240, 28, 28] 960
Hardswish-2 [[1, 240, 28, 28]] [1, 240, 28, 28] 0
Conv2D-26 [[1, 240, 28, 28]] [1, 240, 14, 14] 2,160
BatchNorm2D-20 [[1, 240, 14, 14]] [1, 240, 14, 14] 960
Hardswish-3 [[1, 240, 14, 14]] [1, 240, 14, 14] 0
Conv2D-27 [[1, 240, 14, 14]] [1, 80, 14, 14] 19,200
BatchNorm2D-21 [[1, 80, 14, 14]] [1, 80, 14, 14] 320
InvertedResidual-7 [[1, 40, 28, 28]] [1, 80, 14, 14] 0
Conv2D-28 [[1, 80, 14, 14]] [1, 200, 14, 14] 16,000
BatchNorm2D-22 [[1, 200, 14, 14]] [1, 200, 14, 14] 800
Hardswish-4 [[1, 200, 14, 14]] [1, 200, 14, 14] 0
Conv2D-29 [[1, 200, 14, 14]] [1, 200, 14, 14] 1,800
BatchNorm2D-23 [[1, 200, 14, 14]] [1, 200, 14, 14] 800
Hardswish-5 [[1, 200, 14, 14]] [1, 200, 14, 14] 0
Conv2D-30 [[1, 200, 14, 14]] [1, 80, 14, 14] 16,000
BatchNorm2D-24 [[1, 80, 14, 14]] [1, 80, 14, 14] 320
InvertedResidual-8 [[1, 80, 14, 14]] [1, 80, 14, 14] 0
Conv2D-31 [[1, 80, 14, 14]] [1, 184, 14, 14] 14,720
BatchNorm2D-25 [[1, 184, 14, 14]] [1, 184, 14, 14] 736
Hardswish-6 [[1, 184, 14, 14]] [1, 184, 14, 14] 0
Conv2D-32 [[1, 184, 14, 14]] [1, 184, 14, 14] 1,656
BatchNorm2D-26 [[1, 184, 14, 14]] [1, 184, 14, 14] 736
Hardswish-7 [[1, 184, 14, 14]] [1, 184, 14, 14] 0
Conv2D-33 [[1, 184, 14, 14]] [1, 80, 14, 14] 14,720
BatchNorm2D-27 [[1, 80, 14, 14]] [1, 80, 14, 14] 320
InvertedResidual-9 [[1, 80, 14, 14]] [1, 80, 14, 14] 0
Conv2D-34 [[1, 80, 14, 14]] [1, 184, 14, 14] 14,720
BatchNorm2D-28 [[1, 184, 14, 14]] [1, 184, 14, 14] 736
Hardswish-8 [[1, 184, 14, 14]] [1, 184, 14, 14] 0
Conv2D-35 [[1, 184, 14, 14]] [1, 184, 14, 14] 1,656
BatchNorm2D-29 [[1, 184, 14, 14]] [1, 184, 14, 14] 736
Hardswish-9 [[1, 184, 14, 14]] [1, 184, 14, 14] 0
Conv2D-36 [[1, 184, 14, 14]] [1, 80, 14, 14] 14,720
BatchNorm2D-30 [[1, 80, 14, 14]] [1, 80, 14, 14] 320
InvertedResidual-10 [[1, 80, 14, 14]] [1, 80, 14, 14] 0
Conv2D-37 [[1, 80, 14, 14]] [1, 480, 14, 14] 38,400
BatchNorm2D-31 [[1, 480, 14, 14]] [1, 480, 14, 14] 1,920
Hardswish-10 [[1, 480, 14, 14]] [1, 480, 14, 14] 0
Conv2D-38 [[1, 480, 14, 14]] [1, 480, 14, 14] 4,320
BatchNorm2D-32 [[1, 480, 14, 14]] [1, 480, 14, 14] 1,920
Hardswish-11 [[1, 480, 14, 14]] [1, 480, 14, 14] 0
AdaptiveAvgPool2D-4 [[1, 480, 14, 14]] [1, 480, 1, 1] 0
Conv2D-39 [[1, 480, 1, 1]] [1, 120, 1, 1] 57,720
ReLU-15 [[1, 120, 1, 1]] [1, 120, 1, 1] 0
Conv2D-40 [[1, 120, 1, 1]] [1, 480, 1, 1] 58,080
Hardsigmoid-4 [[1, 480, 1, 1]] [1, 480, 1, 1] 0
SqueezeExcitation-4 [[1, 480, 14, 14]] [1, 480, 14, 14] 0
Conv2D-41 [[1, 480, 14, 14]] [1, 112, 14, 14] 53,760
BatchNorm2D-33 [[1, 112, 14, 14]] [1, 112, 14, 14] 448
InvertedResidual-11 [[1, 80, 14, 14]] [1, 112, 14, 14] 0
Conv2D-42 [[1, 112, 14, 14]] [1, 672, 14, 14] 75,264
BatchNorm2D-34 [[1, 672, 14, 14]] [1, 672, 14, 14] 2,688
Hardswish-12 [[1, 672, 14, 14]] [1, 672, 14, 14] 0
Conv2D-43 [[1, 672, 14, 14]] [1, 672, 14, 14] 6,048
BatchNorm2D-35 [[1, 672, 14, 14]] [1, 672, 14, 14] 2,688
Hardswish-13 [[1, 672, 14, 14]] [1, 672, 14, 14] 0
AdaptiveAvgPool2D-5 [[1, 672, 14, 14]] [1, 672, 1, 1] 0
Conv2D-44 [[1, 672, 1, 1]] [1, 168, 1, 1] 113,064
ReLU-16 [[1, 168, 1, 1]] [1, 168, 1, 1] 0
Conv2D-45 [[1, 168, 1, 1]] [1, 672, 1, 1] 113,568
Hardsigmoid-5 [[1, 672, 1, 1]] [1, 672, 1, 1] 0
SqueezeExcitation-5 [[1, 672, 14, 14]] [1, 672, 14, 14] 0
Conv2D-46 [[1, 672, 14, 14]] [1, 112, 14, 14] 75,264
BatchNorm2D-36 [[1, 112, 14, 14]] [1, 112, 14, 14] 448
InvertedResidual-12 [[1, 112, 14, 14]] [1, 112, 14, 14] 0
Conv2D-47 [[1, 112, 14, 14]] [1, 672, 14, 14] 75,264
BatchNorm2D-37 [[1, 672, 14, 14]] [1, 672, 14, 14] 2,688
Hardswish-14 [[1, 672, 14, 14]] [1, 672, 14, 14] 0
Conv2D-48 [[1, 672, 14, 14]] [1, 672, 7, 7] 16,800
BatchNorm2D-38 [[1, 672, 7, 7]] [1, 672, 7, 7] 2,688
Hardswish-15 [[1, 672, 7, 7]] [1, 672, 7, 7] 0
AdaptiveAvgPool2D-6 [[1, 672, 7, 7]] [1, 672, 1, 1] 0
Conv2D-49 [[1, 672, 1, 1]] [1, 168, 1, 1] 113,064
ReLU-17 [[1, 168, 1, 1]] [1, 168, 1, 1] 0
Conv2D-50 [[1, 168, 1, 1]] [1, 672, 1, 1] 113,568
Hardsigmoid-6 [[1, 672, 1, 1]] [1, 672, 1, 1] 0
SqueezeExcitation-6 [[1, 672, 7, 7]] [1, 672, 7, 7] 0
Conv2D-51 [[1, 672, 7, 7]] [1, 160, 7, 7] 107,520
BatchNorm2D-39 [[1, 160, 7, 7]] [1, 160, 7, 7] 640
InvertedResidual-13 [[1, 112, 14, 14]] [1, 160, 7, 7] 0
Conv2D-52 [[1, 160, 7, 7]] [1, 960, 7, 7] 153,600
BatchNorm2D-40 [[1, 960, 7, 7]] [1, 960, 7, 7] 3,840
Hardswish-16 [[1, 960, 7, 7]] [1, 960, 7, 7] 0
Conv2D-53 [[1, 960, 7, 7]] [1, 960, 7, 7] 24,000
BatchNorm2D-41 [[1, 960, 7, 7]] [1, 960, 7, 7] 3,840
Hardswish-17 [[1, 960, 7, 7]] [1, 960, 7, 7] 0
AdaptiveAvgPool2D-7 [[1, 960, 7, 7]] [1, 960, 1, 1] 0
Conv2D-54 [[1, 960, 1, 1]] [1, 240, 1, 1] 230,640
ReLU-18 [[1, 240, 1, 1]] [1, 240, 1, 1] 0
Conv2D-55 [[1, 240, 1, 1]] [1, 960, 1, 1] 231,360
Hardsigmoid-7 [[1, 960, 1, 1]] [1, 960, 1, 1] 0
SqueezeExcitation-7 [[1, 960, 7, 7]] [1, 960, 7, 7] 0
Conv2D-56 [[1, 960, 7, 7]] [1, 160, 7, 7] 153,600
BatchNorm2D-42 [[1, 160, 7, 7]] [1, 160, 7, 7] 640
InvertedResidual-14 [[1, 160, 7, 7]] [1, 160, 7, 7] 0
Conv2D-57 [[1, 160, 7, 7]] [1, 960, 7, 7] 153,600
BatchNorm2D-43 [[1, 960, 7, 7]] [1, 960, 7, 7] 3,840
Hardswish-18 [[1, 960, 7, 7]] [1, 960, 7, 7] 0
Conv2D-58 [[1, 960, 7, 7]] [1, 960, 7, 7] 24,000
BatchNorm2D-44 [[1, 960, 7, 7]] [1, 960, 7, 7] 3,840
Hardswish-19 [[1, 960, 7, 7]] [1, 960, 7, 7] 0
AdaptiveAvgPool2D-8 [[1, 960, 7, 7]] [1, 960, 1, 1] 0
Conv2D-59 [[1, 960, 1, 1]] [1, 240, 1, 1] 230,640
ReLU-19 [[1, 240, 1, 1]] [1, 240, 1, 1] 0
Conv2D-60 [[1, 240, 1, 1]] [1, 960, 1, 1] 231,360
Hardsigmoid-8 [[1, 960, 1, 1]] [1, 960, 1, 1] 0
SqueezeExcitation-8 [[1, 960, 7, 7]] [1, 960, 7, 7] 0
Conv2D-61 [[1, 960, 7, 7]] [1, 160, 7, 7] 153,600
BatchNorm2D-45 [[1, 160, 7, 7]] [1, 160, 7, 7] 640
InvertedResidual-15 [[1, 160, 7, 7]] [1, 160, 7, 7] 0
Conv2D-62 [[1, 160, 7, 7]] [1, 960, 7, 7] 153,600
BatchNorm2D-46 [[1, 960, 7, 7]] [1, 960, 7, 7] 3,840
Hardswish-20 [[1, 960, 7, 7]] [1, 960, 7, 7] 0
AdaptiveAvgPool2D-9 [[1, 960, 7, 7]] [1, 960, 1, 1] 0
Linear-1 [[1, 960]] [1, 1280] 1,230,080
Hardswish-21 [[1, 1280]] [1, 1280] 0
Dropout-1 [[1, 1280]] [1, 1280] 0
Linear-2 [[1, 1280]] [1, 2] 2,562
===============================================================================
Total params: 4,228,994
Trainable params: 4,180,194
Non-trainable params: 48,800
-------------------------------------------------------------------------------
Input size (MB): 0.57
Forward/backward pass size (MB): 105.41
Params size (MB): 16.13
Estimated Total Size (MB): 122.11
-------------------------------------------------------------------------------
{'total_params': 4228994, 'trainable_params': 4180194}
2.高层 API训练
# 封装模型为一个 model 实例，便于进行后续的训练、评估和推理
model = paddle.Model(mobilenetv3)
# 为模型训练做准备，设置优化器及其学习率，并将网络的参数传入优化器，设置损失函数和精度计算方式
model.prepare(optimizer=paddle.optimizer.Adam(learning_rate=0.001, parameters=model.parameters()),
loss=paddle.nn.CrossEntropyLoss(),
metrics=paddle.metric.Accuracy())
# 启动模型训练，指定训练数据集，设置训练轮次，设置每次数据集计算的批次大小，设置日志格式
model.fit(train_dataset,
test_dataset,
epochs=100,
batch_size=128,
verbose=1)
Epoch 100/100
step 14/14 [==============================] - loss: 0.2047 - acc: 0.9755 - 318ms/step
Eval begin...
step 4/4 [==============================] - loss: 0.2706 - acc: 0.9068 - 221ms/step
Eval samples: 429
3. 模型验证
# 用 evaluate 在测试集上对模型进行验证
eval_result = model.evaluate(test_dataset, verbose=1)
print(eval_result)
Eval begin...
step 429/429 [==============================] - loss: 2.6442 - acc: 0.9138 - 29ms/step
Eval samples: 429
{'loss': [2.6442208], 'acc': 0.9137529137529138}
4.批量预测
# 用 predict 在测试集上对模型进行推理
test_result = model.predict(test_dataset)
# 由于模型是单一输出，test_result的形状为[1, 10000]，10000是测试数据集的数据量。这里打印第一个数据的结果，这个数组表示每个数字的预测概率
print(len(test_result))
Predict begin...
step 429/429 [==============================] - 29ms/step
Predict samples: 429
1
print(test_result[0][0])
# 从测试集中取出一张图片
img, label = test_dataset[0]
print(label)
[[ 5.2088857 -4.8880706]]
0
# 打印推理结果，这里的argmax函数用于取出预测值中概率最高的一个的下标，作为预测标签
import warnings
warnings.filterwarnings('ignore')
pred_label = test_result[0][0].argmax()
print(label)
print(pred_label)
print('true label: {}, pred label: {}'.format(label, pred_label))
# 使用matplotlib库，可视化图片
from matplotlib import pyplot as plt
plt.imshow(img[0])
0
0
true label: 0, pred label: 0
<matplotlib.image.AxesImage at 0x7f88c05daf10>
四、自定义模型
选择自定义CNN模型可以直接跳转至此。
1.自定义卷积模型
# 定义卷积神经网络实现猴痘识别
import paddle.nn as nn
class MyCNN(nn.Layer):
def __init__(self):
super(MyCNN,self).__init__()
self.conv0 = nn.Conv2D(in_channels= 3,out_channels=64, kernel_size=3,stride=1) # 224-3 +1 = 222*222*64
self.pool0 = nn.MaxPool2D(kernel_size=2,stride=2) # 111*111*64
self.conv1 = nn.Conv2D(in_channels = 64,out_channels=128,kernel_size=4,stride = 1) # 111-4+1 = 108*108*128
self.pool1 = nn.MaxPool2D(kernel_size=2,stride=2) # 54*54*128
self.conv2 = nn.Conv2D(in_channels= 128,out_channels=50,kernel_size=5) # 54-5+1 = 50*50*50
self.pool2 = nn.MaxPool2D(kernel_size=2,stride=2) # 25*25*50
self.fc1 = nn.Linear(in_features=50*25*25,out_features=50)
self.fc2 = nn.Linear(in_features=50,out_features=2)
def forward(self,input):
x = self.conv0(input)
x = self.pool0(x)
x = self.conv1(x)
x = self.pool1(x)
x = self.conv2(x)
x = self.pool2(x)
x = paddle.reshape(x,shape=[x.shape[0],-1])
x = self.fc1(x)
y = self.fc2(x)
return y
2.dataloader定义
#训练数据加载
train_loader = paddle.io.DataLoader(train_dataset, batch_size=128, shuffle=True)
#测试数据加载
eval_loader = paddle.io.DataLoader(test_dataset, batch_size = 64, shuffle=False)
3.画图函数定义
import matplotlib.pyplot as plt
%matplotlib inline
Batch=0
Batchs=[]
all_train_accs=[]
def draw_train_acc(Batchs, train_accs):
title="training accs"
plt.title(title, fontsize=24)
plt.xlabel("batch", fontsize=14)
plt.ylabel("acc", fontsize=14)
plt.plot(Batchs, train_accs, color='green', label='training accs')
plt.legend()
plt.grid()
plt.show()
all_train_loss=[]
def draw_train_loss(Batchs, train_loss):
title="training loss"
plt.title(title, fontsize=24)
plt.xlabel("batch", fontsize=14)
plt.ylabel("loss", fontsize=14)
plt.plot(Batchs, train_loss, color='red', label='training loss')
plt.legend()
plt.grid()
plt.show()
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Sized
4.模型训练并保存
# 训练模型-CNN
model=MyCNN() # 模型实例化
model.train() # 训练模式
cross_entropy = paddle.nn.CrossEntropyLoss()
opt=paddle.optimizer.SGD(learning_rate=0.001, parameters=model.parameters())
epochs_num=20 #迭代次数
for pass_num in range(20):
for batch_id,data in enumerate(train_loader()):
image = data[0] #[B,C,H,W]
label = data[1] #[B,1]
label=paddle.unsqueeze(label, 1)
predict=model(image) #数据传入model
loss = cross_entropy(predict,label)
avg_loss = paddle.mean(loss)
acc=paddle.metric.accuracy(predict,label)#计算精度
loss.backward()
opt.step()
opt.clear_grad() #opt.clear_grad()来重置梯度
if batch_id!=0 and batch_id%5==0:
Batch = Batch + 5
Batchs.append(Batch)
all_train_loss.append(loss.numpy()[0])
all_train_accs.append(acc.numpy()[0])
print("epoch:{},step:{},train_loss:{},train_acc:{}".format(pass_num,batch_id,loss.numpy(),acc.numpy()))
paddle.save(model.state_dict(),'MyCNN')#保存模型
epoch:0,step:5,train_loss:[148.6786],train_acc:[0.4375]
epoch:0,step:10,train_loss:[6.2304006],train_acc:[0.3515625]
epoch:1,step:5,train_loss:[0.94077164],train_acc:[0.6171875]
epoch:1,step:10,train_loss:[0.63703007],train_acc:[0.71875]
epoch:2,step:5,train_loss:[0.65847814],train_acc:[0.640625]
epoch:2,step:10,train_loss:[0.6516983],train_acc:[0.6953125]
epoch:3,step:5,train_loss:[0.6700153],train_acc:[0.7578125]
draw_train_acc(Batchs,all_train_accs)
draw_train_loss(Batchs,all_train_loss)
5.模型评估
#模型评估
para_state_dict = paddle.load("MyCNN")
model = MyCNN()
model.set_state_dict(para_state_dict) #加载模型参数
model.eval() #验证模式
accs = []
for batch_id,data in enumerate(eval_loader()):#测试集
image=data[0]
label=data[1]
label=paddle.unsqueeze(label, 1)
predict=model(image)
acc=paddle.metric.accuracy(predict,label)
accs.append(acc.numpy()[0])
avg_acc = np.mean(accs)
print("当前模型在验证集上的准确率为:",avg_acc)
当前模型在验证集上的准确率为: 0.6573909
6.猴痘预测
def load_image(img_path):
'''
预测图片预处理
'''
img = Image.open(img_path)
if img.mode != 'RGB':
img = img.convert('RGB')
img = img.resize((224, 224), Image.BILINEAR)
img = np.array(img).astype('float32')
img = img.transpose((2, 0, 1)) # HWC to CHW
img = img/255 # 像素值归一化
return img
from PIL import Image
para_state_dict = paddle.load("MyCNN")
model = MyCNN()
model.set_state_dict(para_state_dict) #加载模型参数
model.eval() #验证模式
#展示预测图片
infer_path='45-data/Monkeypox/M01_01_10.jpg'
img = Image.open(infer_path)
plt.imshow(img) #根据数组绘制图像
plt.show() #显示图像
#对预测图片进行预处理
infer_imgs = []
infer_imgs.append(load_image(infer_path))
infer_imgs = np.array(infer_imgs)
for i in range(len(infer_imgs)):
data = infer_imgs[i]
dy_x_data = np.array(data).astype('float32')
dy_x_data=dy_x_data[np.newaxis,:, : ,:]
img = paddle.to_tensor (dy_x_data)
out = model(img)
lab = np.argmax(out.numpy()) #argmax():返回最大数的索引
print("第{}个样本,被预测为：{},真实标签为：{}".format(i+1,str(lab),infer_path.split('/')[1]))
ata)
out = model(img)
lab = np.argmax(out.numpy()) #argmax():返回最大数的索引
print("第{}个样本,被预测为：{},真实标签为：{}".format(i+1,str(lab),infer_path.split('/')[1]))
print("结束")
第1个样本,被预测为：1,真实标签为：Monkeypox
结束
此文章为搬运
原项目链接
学大模型，用大模型上飞桨星河社区！每天8点V100G算力免费领！免费领取ERNIE 4.0 100w Token >>>
更多推荐
0
0- 0
已为社区贡献1438条内容
所有评论(0)