【论文复现】多人姿态关键点检测：OpenPose的飞桨复现

0. 导入

跳舞的火柴人

1. 项目简介

该项目给出了Paddle版本的OpenPose实现及其简单应用，该项目主要参照pytorch-openpose，在AI Studio平台上利用Paddle修改并重写了代码，并且完成了模型参数的转化工作。
该项目的内容总结如下：

• 给出了Paddle框架实现的多人姿态关键点提取的网络代码实现，
• 给出了针对15、18、25人体姿态关键点提取模型参数以及人脸部70个关键节点和手部21个关键节点的模型及其参数，
• 给出了使用改代码的实例并对结果做出了可视化展示。

2. OpenPose概述

OpenPose是第一个实时从单张RGB图片提取多人体骨骼框架的系统，该代码库的作者是Ginés Hidalgo, Zhe Cao, Tomas Simon, Shih-En Wei, Yaadhav Raaj, Hanbyul Joo, and Yaser Sheikh。官方代码地址Openpose。Openpose内置了提取人体姿态15、18、25个关键点和人脸部70个关键点、手部21个关键节点的解决方案。下面将以人的姿态关键点为例展开介绍，脸部和手部的关键点类似。

网络结构是一个全卷积的结构，该网络的设计思路在于使用步长大于1的卷积层扩大感受野，使网络能够对人体其他部位有整体的认知，最后的网络输出是包含人体关键节点的热力图，每张热力图表示人的单个节点的位置，网络结构图如下所示。（图片来源：Convolutional Pose Machines）

在文章Convolutional Pose Machines中，网络只能提取单人的pose，为了使它能够泛化到多人姿态的估计，后续的文章Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields提出了人体关键节点关联场（PAF：Part Affinity Fields）的概念，使用PAF关联同一个人的节点，该文章也给出了关联节点后处理步骤的算法。下图给出了PAF的示意图。（图片来源：Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields）

这里只给出了十分简洁的介绍，有兴趣的读者可以到原文中查看细节。推荐阅读顺序

• Convolutional Pose Machines
• Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields
• OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields

3. 代码结构

• /home/aistudio/openpose: 主要代码目录
• /home/aistudio/openpose/models: 模型代码
• /home/aistudio/openpose/postprocess: 后处理代码
• /home/aistudio/openpose/detectors: 主提取关键点类及其方法
• /home/aistudio/openpose/images: 演示实例图片
• /home/aistudio/data/data66164: 内置的模型参数文件

4. 如何使用

4.1 下载依赖包

! cd /home/aistudio/openpose/ && pip install -r ./requirements.txt

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Collecting scikit-image (from -r ./requirements.txt (line 1))
[?25l  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9a/44/8f8c7f9c9de7fde70587a656d7df7d056e6f05192a74491f7bc074a724d0/scikit_image-0.19.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (13.3MB)
[K     |████████████████████████████████| 13.3MB 7.7MB/s eta 0:00:01
[?25hRequirement already satisfied: pillow!=7.1.0,!=7.1.1,!=8.3.0,>=6.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r ./requirements.txt (line 1)) (7.1.2)
Requirement already satisfied: imageio>=2.4.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r ./requirements.txt (line 1)) (2.6.1)
Requirement already satisfied: scipy>=1.4.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r ./requirements.txt (line 1)) (1.6.3)
Requirement already satisfied: networkx>=2.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r ./requirements.txt (line 1)) (2.4)
Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r ./requirements.txt (line 1)) (21.3)
Collecting PyWavelets>=1.1.1 (from scikit-image->-r ./requirements.txt (line 1))
[?25l  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a1/9c/564511b6e1c4e1d835ed2d146670436036960d09339a8fa2921fe42dad08/PyWavelets-1.2.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (6.1MB)
[K     |████████████████████████████████| 6.2MB 9.0MB/s eta 0:00:01
[?25hCollecting tifffile>=2019.7.26 (from scikit-image->-r ./requirements.txt (line 1))
[?25l  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d8/38/85ae5ed77598ca90558c17a2f79ddaba33173b31cf8d8f545d34d9134f0d/tifffile-2021.11.2-py3-none-any.whl (178kB)
[K     |████████████████████████████████| 184kB 56.4MB/s eta 0:00:01
[?25hRequirement already satisfied: numpy>=1.17.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r ./requirements.txt (line 1)) (1.20.3)
Requirement already satisfied: decorator>=4.3.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from networkx>=2.2->scikit-image->-r ./requirements.txt (line 1)) (4.4.2)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from packaging>=20.0->scikit-image->-r ./requirements.txt (line 1)) (3.0.6)
Installing collected packages: PyWavelets, tifffile, scikit-image
Successfully installed PyWavelets-1.2.0 scikit-image-0.19.1 tifffile-2021.11.2

4.2 创建网络，加载参数，读入数据

import sys
sys.path.append('/home/aistudio/openpose')
import cv2
from openpose.detectors.detector import PoseDetector

detector = PoseDetector( pose_points=25, detect_face=True, detect_hand=True)
oriImg = cv2.imread('/home/aistudio/openpose/images/COCO_val2014_000000000241.jpg')
output = detector(oriImg)

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layers/utils.py:26: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  def convert_to_list(value, n, name, dtype=np.int):
W0112 21:14:26.214571  3289 device_context.cc:362] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W0112 21:14:26.219558  3289 device_context.cc:372] device: 0, cuDNN Version: 7.6.
/home/aistudio/openpose/detectors/detector.py:68: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  body_keypoints = np.array(candidate)[np.array(person[:-2]).astype(np.int)][:, :2]
/home/aistudio/openpose/detectors/detector.py:69: DeprecationWarning: `np.int` is a deprecated alias for the builtin `int`. To silence this warning, use `int` by itself. Doing this will not modify any behavior and is safe. When replacing `np.int`, you may wish to use e.g. `np.int64` or `np.int32` to specify the precision. If you wish to review your current use, check the release note link for additional information.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  body_keypoints[np.array(person[:-2]).astype(np.int) == -1] *= -1

4.3 可视化展示结果

from openpose.util import draw_pose
import matplotlib.pyplot as plt

canvas = draw_pose(oriImg, output, pose_points=25)

plt.rcParams['figure.dpi'] = 150
plt.axis('off')
plt.imshow(canvas[:, :, [2, 1, 0]])

/home/aistudio/openpose/util.py:77: DeprecationWarning: elementwise comparison failed; this will raise an error in the future.
  if face_keypoints != []:
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2349: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  if isinstance(obj, collections.Iterator):
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2366: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  return list(data) if isinstance(data, collections.MappingView) else data





<matplotlib.image.AxesImage at 0x7f680c4a4290>

4.4 其他模型

• 检测人体25个关键节点

import sys
sys.path.append('/home/aistudio/openpose')

from openpose.detectors.body import Body

body_estimation = Body(25)
oriImg = cv2.imread('/home/aistudio/openpose/images/COCO_val2014_000000000241.jpg')
candidate, subset = body_estimation(oriImg)

import cv2
import matplotlib.pyplot as plt
from openpose.util import draw_bodypose

canvas = draw_bodypose(oriImg, candidate, subset, 25)
plt.rcParams['figure.dpi'] = 150
plt.axis('off')
plt.imshow(canvas[:, :, [2, 1, 0]])

<matplotlib.image.AxesImage at 0x7f680c288950>

• 检测手部关键节点

import sys
sys.path.append('/home/aistudio/openpose')

from openpose.detectors.hand import Hand


hand_estimation = Hand()
oriImg = cv2.imread('/home/aistudio/openpose/images/hand.jpg')
output = hand_estimation(oriImg)

import cv2
import matplotlib.pyplot as plt
from openpose.util import draw_handpose

canvas = draw_handpose(oriImg, [output])
plt.rcParams['figure.dpi'] = 150
plt.axis('off')
plt.imshow(canvas[:, :, [2, 1, 0]])

<matplotlib.image.AxesImage at 0x7f680c332150>

• 检测脸部关键节点

import sys
sys.path.append('/home/aistudio/openpose')
import cv2

from openpose.detectors.face import Face

face_estimation = Face()
oriImg = cv2.imread('/home/aistudio/openpose/images/demo.jpg')
output = face_estimation(oriImg)

from util import draw_facepose
import cv2
import matplotlib.pyplot as plt

canvas = draw_facepose(oriImg, [output])
plt.rcParams['figure.dpi'] = 150
plt.axis('off')
plt.imshow(canvas[:, :, [2, 1, 0]])

<matplotlib.image.AxesImage at 0x7f680c07bc10>

4.5 可视化展示PAF

import sys
sys.path.append('/home/aistudio/openpose')
import cv2
import numpy as np

import paddle.fluid as F
import paddle.fluid.dygraph as dg

from openpose.models import build_body_model

body_model = build_body_model(18)
body_model.eval()

oriImg = cv2.imread('/home/aistudio/openpose/images/COCO_val2014_000000000241.jpg')
oriImg = np.float32(oriImg[:, :, :, np.newaxis]).transpose((3, 2, 0, 1)) / 255. - 0.5
oriImg = dg.to_variable(oriImg)

output = body_model(oriImg)

import matplotlib.pyplot as plt

plt.rcParams['figure.dpi'] = 150
plt.rcParams['figure.figsize'] = (5, 4)
for stage in range(6):
    plt.subplot(2, 3, stage + 1)
    plt.axis('off')
    plt.title("stage%d" % (stage + 1))
    plt.imshow(output[stage].numpy()[0][:18].sum(axis=0))

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/image.py:425: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  a_min = np.asscalar(a_min.astype(scaled_dtype))
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/image.py:426: DeprecationWarning: np.asscalar(a) is deprecated since NumPy v1.16, use a.item() instead
  a_max = np.asscalar(a_max.astype(scaled_dtype))

import matplotlib.pyplot as plt

plt.rcParams['figure.dpi'] = 150
plt.rcParams['figure.figsize'] = (5, 4)
for stage in range(6):
    plt.subplot(2, 3, stage + 1)
    plt.axis('off')
    plt.title("stage%d" % (stage + 1))
 plt.subplot(2, 3, stage + 1)
    plt.axis('off')
    plt.title("stage%d" % (stage + 1))
    plt.imshow(output[stage].numpy()[0][19])

About this code

Several pose estimation model written in Paddle (No code for training). This code is heavily copied from pytorch-openpose.

Original project: Openpose

Paper: OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields

If this work is useful for you, please cite

@article{8765346,
author = {Z. {Cao} and G. {Hidalgo Martinez} and T. {Simon} and S. {Wei} and Y. A. {Sheikh}},
journal = {IEEE Transactions on Pattern Analysis and Machine Intelligence},
title = {OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
year = {2019}
}

@inproceedings{simon2017hand,
author = {Tomas Simon and Hanbyul Joo and Iain Matthews and Yaser Sheikh},
booktitle = {CVPR},
title = {Hand Keypoint Detection in Single Images using Multiview Bootstrapping},
year = {2017}
}

@inproceedings{cao2017realtime,
author = {Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh},
booktitle = {CVPR},
title = {Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields},
year = {2017}
}

@inproceedings{wei2016cpm,
author = {Shih-En Wei and Varun Ramakrishna and Takeo Kanade and Yaser Sheikh},
booktitle = {CVPR},
title = {Convolutional pose machines},
year = {2016}
}