AI元宇宙敲门砖【1】：可微神经渲染助力虚拟物体创造

---

引入

• 元宇宙：元宇宙（Metaverse）是利用科技手段进行链接与创造的，与现实世界映射与交互的虚拟世界，具备新型社会体系的数字生活空间。
• 三维模型：三维模型是物体的多边形表示，通常用计算机或者其它视频设备进行显示。显示的物体可以是现实世界的实体，也可以是虚构的物体。任何物理自然界存在的东西都可以用三维模型表示。三维模型的渲染显示在元宇宙中可谓是视觉方面的重中之重。先有看的发展，才会逐步发展到听、嗅、味、触等感官出现在元宇宙中。
• 可微渲染：简单的说，可微渲染就是计算渲染过程的导数。其具体的应用可以包括解决inverse rendering以及实现将渲染过程放入神经网络中以解决更复杂的视觉问题（可导性是训练网络的必要条件）。可微代表着可以求梯度，于是可以端到端的反向推导出图片对应的三维模型。
• Mesh：三维模型的多边形网格表示。使用大量空间三角形或多边形来代表物体的各个面。这种三维模型的表示方法是比较节省资源的。其它常见的表示方法有：体素表示、点云表示。因为Mesh占用存储也比较少，所以在硬件还未跟上的情况下，最适合用于元宇宙中的三维表示。

Neural 3D Mesh Renderer 简介

对于二维图像背后的三维世界建模，哪一种三维表示最合适？现在常见的表示方法有点云、体素和网格，其中多边形网格具有良好的紧致性和几何性质。但是使用神经网络直接由多边形网格生成图像比较困难，因为光栅化的过程阻止了反向传播。

因此，本项目所参考的Repo的原作者提出了一个近似的梯度栅格化，使渲染集成到神经网络。使用这个渲染器，可以执行带有轮廓图像监督的单图像三维网格重建，并且该系统比现有的基于体素的方法更好。

利用此类可微分的多边形网格光栅化与渲染，才有机会在元宇宙三维世界中进行视觉上的物体生成。

实现

neural-renderer-paddle。欢迎来点Star哦！

动手实践

准备代码与运行环境

安装笔者预先准备好的包文件，完成环境配置

# 安装命令
# git clone https://github.com/AgentMaker/neural-renderer-paddle
%pip install neural-renderer-paddle/

Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Processing ./neural-renderer-paddle
Collecting scikit-image~=0.19.0 (from neural-renderer-paddle==1.1.3.3)
[?25l  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ae/3d/b2662293ff78c5a500098fb1edb23ffa7c3dcd39da8bfdf0e74a8f2a1bdb/scikit_image-0.19.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (55.4MB)
[K     |████████████████████████████████| 55.4MB 588kB/s  eta 0:00:01
[?25hRequirement already satisfied: paddlepaddle-gpu~=2.2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from neural-renderer-paddle==1.1.3.3) (2.2.1.post101)
Requirement already satisfied: pillow!=7.1.0,!=7.1.1,!=8.3.0,>=6.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image~=0.19.0->neural-renderer-paddle==1.1.3.3) (7.1.2)
Collecting PyWavelets>=1.1.1 (from scikit-image~=0.19.0->neural-renderer-paddle==1.1.3.3)
[?25l  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a1/9c/564511b6e1c4e1d835ed2d146670436036960d09339a8fa2921fe42dad08/PyWavelets-1.2.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (6.1MB)
[K     |████████████████████████████████| 6.2MB 11.4MB/s eta 0:00:01
[?25hRequirement already satisfied: numpy>=1.17.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image~=0.19.0->neural-renderer-paddle==1.1.3.3) (1.20.3)
Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image~=0.19.0->neural-renderer-paddle==1.1.3.3) (20.9)
Requirement already satisfied: scipy>=1.4.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image~=0.19.0->neural-renderer-paddle==1.1.3.3) (1.6.3)
Collecting tifffile>=2019.7.26 (from scikit-image~=0.19.0->neural-renderer-paddle==1.1.3.3)
[?25l  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d8/38/85ae5ed77598ca90558c17a2f79ddaba33173b31cf8d8f545d34d9134f0d/tifffile-2021.11.2-py3-none-any.whl (178kB)
[K     |████████████████████████████████| 184kB 5.1MB/s eta 0:00:01
[?25hRequirement already satisfied: networkx>=2.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image~=0.19.0->neural-renderer-paddle==1.1.3.3) (2.4)
Requirement already satisfied: imageio>=2.4.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image~=0.19.0->neural-renderer-paddle==1.1.3.3) (2.6.1)
Requirement already satisfied: decorator in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu~=2.2.0->neural-renderer-paddle==1.1.3.3) (4.4.2)
Requirement already satisfied: protobuf>=3.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu~=2.2.0->neural-renderer-paddle==1.1.3.3) (3.14.0)
Requirement already satisfied: six in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu~=2.2.0->neural-renderer-paddle==1.1.3.3) (1.15.0)
Requirement already satisfied: astor in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu~=2.2.0->neural-renderer-paddle==1.1.3.3) (0.8.1)
Requirement already satisfied: requests>=2.20.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlepaddle-gpu~=2.2.0->neural-renderer-paddle==1.1.3.3) (2.22.0)
Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from packaging>=20.0->scikit-image~=0.19.0->neural-renderer-paddle==1.1.3.3) (2.4.2)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu~=2.2.0->neural-renderer-paddle==1.1.3.3) (1.25.6)
Requirement already satisfied: idna<2.9,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu~=2.2.0->neural-renderer-paddle==1.1.3.3) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu~=2.2.0->neural-renderer-paddle==1.1.3.3) (2019.9.11)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.20.0->paddlepaddle-gpu~=2.2.0->neural-renderer-paddle==1.1.3.3) (3.0.4)
Building wheels for collected packages: neural-renderer-paddle
  Building wheel for neural-renderer-paddle (setup.py) ... [?25ldone
[?25h  Created wheel for neural-renderer-paddle: filename=neural_renderer_paddle-1.1.3.3-cp37-none-any.whl size=31171 sha256=376d0112f2cc63893da40226f8029f38c40bfb1a3ff64e36d3553fe610f89820
  Stored in directory: /home/aistudio/.cache/pip/wheels/7d/a9/a3/fdbdc97716f908818b827591eecb4f42348bf721f3089d8201
Successfully built neural-renderer-paddle
Installing collected packages: PyWavelets, tifffile, scikit-image, neural-renderer-paddle
Successfully installed PyWavelets-1.2.0 neural-renderer-paddle-1.1.3.3 scikit-image-0.19.0 tifffile-2021.11.2
Note: you may need to restart the kernel to use updated packages.

进行示例项目的实验

• 进入到笔者准备的示例文件夹【neural-renderer-paddle/examples】，示例源码都在里面

%cd ~/neural-renderer-paddle/examples/

/home/aistudio/neural-renderer-paddle/examples

• 导入示例函数

import warnings
warnings.filterwarnings("ignore")

from example1 import main as example1
from example2 import main as example2
from example3 import main as example3
from example4 import main as example4

/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/__init__.py:107: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import MutableMapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/rcsetup.py:20: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Iterable, Mapping
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/colors.py:53: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Sized


Compiling user custom op, it will cost a few seconds.....


cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layers/utils.py:77: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  return (isinstance(seq, collections.Sequence) and


Compiling user custom op, it will cost a few seconds.....


cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layers/utils.py:77: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  return (isinstance(seq, collections.Sequence) and
W1211 11:00:06.778892   400 custom_operator.cc:522] Operator (load_textures)has been registered.


Compiling user custom op, it will cost a few seconds.....


cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/fluid/layers/utils.py:77: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  return (isinstance(seq, collections.Sequence) and
W1211 11:00:13.350206   400 custom_operator.cc:522] Operator (backward_depth_map)has been registered.
W1211 11:00:13.350270   400 custom_operator.cc:522] Operator (forward_texture_sampling)has been registered.
W1211 11:00:13.350276   400 custom_operator.cc:522] Operator (backward_pixel_map)has been registered.
W1211 11:00:13.350281   400 custom_operator.cc:522] Operator (load_textures)has been registered.
W1211 11:00:13.350286   400 custom_operator.cc:522] Operator (forward_face_index_map)has been registered.
W1211 11:00:13.350289   400 custom_operator.cc:522] Operator (backward_textures)has been registered.

• 样例1：朴素地渲染一个茶壶模型

example1()

W1211 11:07:30.223091   400 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W1211 11:07:30.228544   400 device_context.cc:465] device: 0, cuDNN Version: 7.6.
Drawing:   0%|          | 0/90 [00:00<?, ?it/s]/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/tensor/creation.py:130: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
Drawing: 100%|██████████| 90/90 [00:02<00:00, 30.09it/s]

# 显示渲染结果
from IPython.display import display, Image

display(Image('data/example1.gif'))

<IPython.core.display.Image object>

• 样例2：将茶壶模型的指定视角拟合为正方形

example2()

Optimizing:   0%|          | 0/300 [00:00<?, ?it/s]/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/tensor/creation.py:130: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
Optimizing: 100%|██████████| 300/300 [00:17<00:00, 16.71it/s]
Drawing: 100%|██████████| 90/90 [00:02<00:00, 32.14it/s]

from IPython.display import display, Image

print('拟合过程：')
display(Image('data/example2_optimization.gif'))

print('拟合结果：')
display(Image('data/example2_result.gif'))

拟合过程：



<IPython.core.display.Image object>


拟合结果：



<IPython.core.display.Image object>

• 样例3：为茶壶模型上色纹理

example3()

Optimizing:   0%|          | 0/300 [00:00<?, ?it/s]/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/tensor/creation.py:130: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
Optimizing: 100%|██████████| 300/300 [00:17<00:00, 17.55it/s]
Drawing: 100%|██████████| 90/90 [00:05<00:00, 13.89it/s]

from IPython.display import display, Image

print('纹理参考图片：')
display(Image('data/example3_ref.png'))

print('纹理拟合结果：')
display(Image('data/example3_result.gif'))

纹理参考图片：

纹理拟合结果：



<IPython.core.display.Image object>

• 样例4：拟合茶壶模型到未知视角，从而解出视角值

example4()

  0%|          | 0/1000 [00:00<?, ?it/s]/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/paddle/tensor/creation.py:130: DeprecationWarning: `np.object` is a deprecated alias for the builtin `object`. To silence this warning, use `object` by itself. Doing this will not modify any behavior and is safe. 
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  if data.dtype == np.object:
Optimizing (loss 69.2500):  15%|█▍        | 148/1000 [00:07<00:49, 17.35it/s] 

from IPython.display import display, Image

print('初始视角：')
display(Image('data/example4_init.png'))

print('欲拟合的未知视角：')
display(Image('data/example4_ref.png'))

print('拟合过程：')
display(Image('data/example4_result.gif'))

初始视角：

欲拟合的未知视角：

拟合过程：



<IPython.core.display.Image object>

结语

Neural 3D Mesh Renderer 可谓是开启了深度学习的新时代，在它出现之后，图形学也逐渐走进了深度学习的世界。

元宇宙的世界，就如同是游戏的世界，进行了一个概念的拓展。所以AI在未来元宇宙的布局中肯定是必不可少的。

可微渲染结合参数化三维人脸模型FLAME可以实现AI智能捏脸，若需要更进一步，使用SMPL-X，则有可能进行全身的AI建模。

而要实现这些，本篇项目便是大的基础。不过因为是基础，仍需要很多的建设，本项目还很难实现制作元宇宙虚拟形象的功能。大家可以期待笔者未来在此类项目中的投入。

最后，欢迎给我们团队的neural-renderer-paddle仓库点个Star。😀