足球精彩时刻剪辑

转载自AI Studio

项目链接https://aistudio.baidu.com/aistudio/projectdetail/3473391?contributionType=1

1.项目概述

近期的全民热点话题离不开正在举办的冬奥会，每天都有赛场上的喜讯传来，谷爱凌夺中国第三金、武大靖和任子威的亮眼表现、苏翊鸣的“飞檐走壁”，可喜可贺！

本届冬奥会共有7个大项、15个分项和109个小项，每场比赛结束都有很多记录运动员比赛精彩瞬间的回放视频产生，这些短视频通常播放量很高，深受大众喜爱。除了服务于大众，让体育爱好者在欣赏时大饱眼福外，精彩视频剪辑技术也可以用于专业的体育训练，运动员可以通过比赛或日常训练回放来熟悉自己和对手，进行战术演练，达到知己知彼百战百胜的效果。

本项目运用动作检测技术，实现基于足球视频的高精度精彩视频剪辑方案，最终方案动作识别准确率达91%，动作检测F1-score达到76.2%。快一起来看看吧～

图1 足球精彩视频剪辑示例

欢迎报名直播课加入交流群，如需更多技术交流与合作可点击以下链接
https://paddleqiyeban.wjx.cn/vj/Qlb0uS3.aspx?udsid=531417

2.技术难点

目前精彩视频剪辑虽然需求量大应用广泛，但人工剪辑需要浏览整个比赛视频，工作量大、成本高，又要求剪辑人员同时具有一定的体育专业知识，专业需求高。如果用AI技术来实现会存在以下两个难点问题：

  1. 动作检测任务复杂度高：视频精彩片段剪辑任务的实现要点在于准确找到该类动作发生的起止点。但体育类视频内经常包含大量冗余的背景信息，动作类别多样且持续时长相对较短，要精准的判断出动作的起始点和对应的类别，任务难度高。
  2. 视频中的信息具有多样性，如何有效利用这些特征信息也是值得我们去考虑的。

3.解决方案

动作检测任务可以理解为动作定位+识别任务，需要在一个长视频中，先定位到一个动作发生的起止点，再识别出这个动作是什么。本案例使用PP-TSM+BMN+AttentionLSTM系列模型来实现动作检测任务，具体流程如下图：

图2 动作检测任务流程

整个方案可以分为三个步骤：

  1. 使用飞桨特色的高精度视频理解模型PP-TSM提取视频图像特征；使用VGGish网络提取音频特征；
  2. 将获得的音视频特征输入BMN网络，得到由动作开始时间和结束时间组合成的时序片段（proposal）；
  3. 得到时序片段提名后，根据动作开始和结束时间截断视频和音频特征，通过AttentionLSTM输出动作类别。

4.数据准备

4.1 数据集介绍

数据集由EuroCup2012, EuroCup2016, WorldCup2014, WorldCup2018四个赛事的比赛视频组成，共计272个训练集、25个测试集，支持15种足球精彩动作定位与识别，动作类别分别为：射门、进球、进球有欢呼、角球、任意球、黄牌、红牌、点球、换人、界外球、球门球、开球、越位挥旗、回放空中对抗和回放进球。本案例中我们提供一条视频数据供大家进行测试，可以通过如下命令，下载视频。

# 仅在第一次运行notebook时，执行以下命令
!wget https://bj.bcebos.com/v1/tmt-pub/datasets/EuroCup2016/63e51df254d2402fac703b6c4fdb4ea9.mp4
!mv 63e51df254d2402fac703b6c4fdb4ea9.mp4 football.mp4

--2022-02-18 14:00:03--  https://bj.bcebos.com/v1/tmt-pub/datasets/EuroCup2016/63e51df254d2402fac703b6c4fdb4ea9.mp4
正在解析主机 bj.bcebos.com (bj.bcebos.com)... 182.61.200.195, 182.61.200.229, 2409:8c04:1001:1002:0:ff:b001:368a
正在连接 bj.bcebos.com (bj.bcebos.com)|182.61.200.195|:443... 已连接。
已发出 HTTP 请求，正在等待回应... 200 OK
长度： 1562942061 (1.5G) [video/mp4]
正在保存至: “63e51df254d2402fac703b6c4fdb4ea9.mp4”

63e51df254d2402fac7 100%[===================>]   1.46G  89.6MB/s    in 20s     

2022-02-18 14:00:23 (75.0 MB/s) - 已保存 “63e51df254d2402fac703b6c4fdb4ea9.mp4” [1562942061/1562942061])

除视频文件外，数据集中还包含视频对应的标注文件，这里我们在PaddleVideo/applications/FootballAction/datasets/EuroCup2016 文件路径下提供了标注文件 label.json。标注格式为：

{
    "fps": 5,
    "gts": [
        {
            "url": "football.mp4",
            "total_frames": 6189,
            "actions": [
                {
                    "label_ids": [
                        2
                    ],
                    "label_names": [
                        "角球"
                    ],
                    "start_id": 393,
                    "end_id": 399
                },
                ...

文件中的 fps：5 代表以每秒5帧的频率提取视频帧。
actions 包含每段视频中的动作信息：动作类别标签（label_ids）、类别名称（label_names）、开始帧数（start_id）和结束帧数（end_id）。

该视频样本包含8种动作信息，动作类别标签与类别名称的对应关系如下：

    "0": "背景"
    "1": "进球"
    "2": "角球"
    "3": "任意球"
    "4": "黄牌"
    "5": "红牌"
    "6": "换人"
    "7": "界外球"

4.2 视频采样

输入视频为mp4文件，我们提供的视频样本 football.mp4 时长1h43min。训练时如果使用全部视频文件，会消耗大量计算资源，一般预先做一些采样处理。

图像采样：以fps=5的频率抽取图像帧
音频采样：pcm音频文件，采样频率ar=16000

运行以下代码进行图像和音频采样。

# 移动视频football.mp4、label.json到指定位置
%cd /home/aistudio/PaddleVideo/applications/FootballAction/datasets/EuroCup2016/
!mkdir mp4
!mv /home/aistudio/football.mp4 /home/aistudio/PaddleVideo/applications/FootballAction/datasets/EuroCup2016/mp4/football.mp4

/home/aistudio/PaddleVideo/applications/FootballAction/datasets/EuroCup2016

# 开始运行代码前先安装好依赖
%cd /home/aistudio/PaddleVideo/
!python -m pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple/

/home/aistudio/PaddleVideo
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple/
Requirement already satisfied: numpy in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 1)) (1.19.5)
Requirement already satisfied: pandas in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.1.5)
Requirement already satisfied: tqdm in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (4.27.0)
Requirement already satisfied: PyYAML>=5.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (5.1.2)
Collecting opencv-python==4.2.0.32
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/34/a3/403dbaef909fee9f9f6a8eaff51d44085a14e5bb1a1ff7257117d744986a/opencv_python-4.2.0.32-cp37-cp37m-manylinux1_x86_64.whl (28.2 MB)
     |████████████████████████████████| 28.2 MB 9.3 MB/s            
[?25hCollecting decord==0.4.2
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c0/0c/7d99cfcde7b85f80c9ea9b0b19441339ad3cef59ee7fa5386598db714efe/decord-0.4.2-py2.py3-none-manylinux1_x86_64.whl (11.8 MB)
     |████████████████████████████████| 11.8 MB 444 kB/s            
[?25hCollecting av==8.0.3
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/66/ff/bacde7314c646a2bd2f240034809a10cc3f8b096751284d0828640fff3dd/av-8.0.3-cp37-cp37m-manylinux2010_x86_64.whl (37.2 MB)
     |████████████████████████████████| 37.2 MB 10.9 MB/s            
[?25hRequirement already satisfied: scipy in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (1.6.3)
Collecting ffmpeg-python==0.2.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d7/0c/56be52741f75bad4dc6555991fabd2e07b432d333da82c11ad701123888a/ffmpeg_python-0.2.0-py3-none-any.whl (25 kB)
Collecting scikit-image
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d2/d9/d16d4cbb4840e0fb3bd329b49184d240b82b649e1bd579489394fbc85c81/scikit_image-0.19.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (13.5 MB)
     |████████████████████████████████| 13.5 MB 5.9 MB/s            
[?25hRequirement already satisfied: matplotlib in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (2.2.3)
Requirement already satisfied: paddlenlp in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (2.1.1)
Collecting lmdb
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/4d/cf/3230b1c9b0bec406abb85a9332ba5805bdd03a1d24025c6bbcfb8ed71539/lmdb-1.3.0-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (298 kB)
     |████████████████████████████████| 298 kB 10.9 MB/s            
[?25hRequirement already satisfied: moviepy in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.0.1)
Collecting SimpleITK
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ad/ce/75d95eab0bc815f478b594add92090ff664e5cbc75ab9d8e2ffbaae1dc91/SimpleITK-2.1.1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (48.4 MB)
     |████████████████████████████████| 48.4 MB 5.7 MB/s            
[?25hCollecting paddledet
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/bb/27/eb6714bdc12e73544617b1b4b3481ae70bec644a722bcbedd5743531db97/paddledet-2.3.0-py3-none-any.whl (589 kB)
     |████████████████████████████████| 589 kB 6.2 MB/s            
[?25hRequirement already satisfied: et_xmlfile in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (1.0.1)
Collecting lap
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/bf/64/d9fb6a75b15e783952b2fec6970f033462e67db32dc43dfbb404c14e91c2/lap-0.4.0.tar.gz (1.5 MB)
     |████████████████████████████████| 1.5 MB 7.8 MB/s            
[?25h  Preparing metadata (setup.py) ... [?25ldone
[?25hCollecting motmetrics
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9c/28/9c3bc8e2a87f4c9e7b04ab72856ec7f9895a66681a65973ffaf9562ef879/motmetrics-1.2.0-py3-none-any.whl (151 kB)
     |████████████████████████████████| 151 kB 4.4 MB/s            
[?25hCollecting xmltodict
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/28/fd/30d5c1d3ac29ce229f6bdc40bbc20b28f716e8b363140c26eff19122d8a5/xmltodict-0.12.0-py2.py3-none-any.whl (9.2 kB)
Requirement already satisfied: openpyxl in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from -r requirements.txt (line 21)) (3.0.5)
Requirement already satisfied: future in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from ffmpeg-python==0.2.0->-r requirements.txt (line 9)) (0.18.0)
Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pandas->-r requirements.txt (line 2)) (2.8.2)
Requirement already satisfied: pytz>=2017.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pandas->-r requirements.txt (line 2)) (2019.3)
Collecting PyWavelets>=1.1.1
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/a1/9c/564511b6e1c4e1d835ed2d146670436036960d09339a8fa2921fe42dad08/PyWavelets-1.2.0-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.whl (6.1 MB)
     |████████████████████████████████| 6.1 MB 3.6 MB/s            
[?25hCollecting tifffile>=2019.7.26
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d8/38/85ae5ed77598ca90558c17a2f79ddaba33173b31cf8d8f545d34d9134f0d/tifffile-2021.11.2-py3-none-any.whl (178 kB)
     |████████████████████████████████| 178 kB 6.2 MB/s            
[?25hRequirement already satisfied: pillow!=7.1.0,!=7.1.1,!=8.3.0,>=6.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r requirements.txt (line 10)) (8.2.0)
Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r requirements.txt (line 10)) (21.3)
Requirement already satisfied: imageio>=2.4.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r requirements.txt (line 10)) (2.6.1)
Requirement already satisfied: networkx>=2.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-image->-r requirements.txt (line 10)) (2.4)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->-r requirements.txt (line 11)) (1.1.0)
Requirement already satisfied: six>=1.10 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->-r requirements.txt (line 11)) (1.16.0)
Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->-r requirements.txt (line 11)) (0.10.0)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->-r requirements.txt (line 11)) (3.0.7)
Requirement already satisfied: colorama in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp->-r requirements.txt (line 12)) (0.4.4)
Requirement already satisfied: colorlog in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp->-r requirements.txt (line 12)) (4.1.0)
Requirement already satisfied: h5py in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp->-r requirements.txt (line 12)) (2.9.0)
Requirement already satisfied: multiprocess in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp->-r requirements.txt (line 12)) (0.70.11.1)
Requirement already satisfied: paddlefsl==1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp->-r requirements.txt (line 12)) (1.0.0)
Requirement already satisfied: jieba in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp->-r requirements.txt (line 12)) (0.42.1)
Requirement already satisfied: seqeval in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp->-r requirements.txt (line 12)) (1.2.2)
Requirement already satisfied: requests~=2.24.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlefsl==1.0.0->paddlenlp->-r requirements.txt (line 12)) (2.24.0)
Requirement already satisfied: decorator<5.0,>=4.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from moviepy->-r requirements.txt (line 14)) (4.4.2)
Requirement already satisfied: imageio-ffmpeg>=0.2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from moviepy->-r requirements.txt (line 14)) (0.3.0)
Requirement already satisfied: proglog<=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from moviepy->-r requirements.txt (line 14)) (0.1.9)
Requirement already satisfied: Cython in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddledet->-r requirements.txt (line 16)) (0.29)
Collecting cython-bbox
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/fa/b9/fc7d60e8c3b29cc0ff24a3bb3c4b7457e10b7610fbb2893741b623487b34/cython_bbox-0.1.3.tar.gz (41 kB)
     |████████████████████████████████| 41 kB 613 kB/s             
[?25h  Preparing metadata (setup.py) ... [?25ldone
[?25hRequirement already satisfied: sklearn in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddledet->-r requirements.txt (line 16)) (0.0)
Requirement already satisfied: visualdl>=2.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddledet->-r requirements.txt (line 16)) (2.2.0)
Requirement already satisfied: setuptools>=42.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddledet->-r requirements.txt (line 16)) (56.2.0)
Collecting typeguard
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9a/bb/d43e5c75054e53efce310e79d63df0ac3f25e34c926be5dffb7d283fb2a8/typeguard-2.13.3-py3-none-any.whl (17 kB)
Collecting terminaltables
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/c4/fb/ea621e0a19733e01fe4005d46087d383693c0f4a8f824b47d8d4122c87e0/terminaltables-3.1.10-py2.py3-none-any.whl (15 kB)
Collecting shapely
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9d/4d/4b0d86ed737acb29c5e627a91449470a9fb914f32640db3f1cb7ba5bc19e/Shapely-1.8.1.post1-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.0 MB)
     |████████████████████████████████| 2.0 MB 5.1 MB/s            
[?25hCollecting pycocotools
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/75/5c/ac61ea715d7a89ecc31c090753bde28810238225ca8b71778dfe3e6a68bc/pycocotools-2.0.4.tar.gz (106 kB)
     |████████████████████████████████| 106 kB 434 kB/s            
[?25h  Installing build dependencies ... [?25ldone
[?25h  Getting requirements to build wheel ... [?25ldone
[?25h  Preparing metadata (pyproject.toml) ... [?25ldone
[?25hCollecting pytest-benchmark
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2c/60/423a63fb190a0483d049786a121bd3dfd7d93bb5ff1bb5b5cd13e5df99a7/pytest_benchmark-3.4.1-py2.py3-none-any.whl (50 kB)
     |████████████████████████████████| 50 kB 217 kB/s             
[?25hRequirement already satisfied: flake8 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from motmetrics->-r requirements.txt (line 19)) (4.0.1)
Collecting pytest
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/38/93/c7c0bd1e932b287fb948eb9ce5a3d6307c9fc619db1e199f8c8bc5dad95f/pytest-7.0.1-py3-none-any.whl (296 kB)
     |████████████████████████████████| 296 kB 4.0 MB/s            
[?25hCollecting flake8-import-order
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ab/52/cf2d6e2c505644ca06de2f6f3546f1e4f2b7be34246c9e0757c6048868f9/flake8_import_order-0.18.1-py2.py3-none-any.whl (15 kB)
Requirement already satisfied: jdcal in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from openpyxl->-r requirements.txt (line 21)) (1.4.1)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests~=2.24.0->paddlefsl==1.0.0->paddlenlp->-r requirements.txt (line 12)) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests~=2.24.0->paddlefsl==1.0.0->paddlenlp->-r requirements.txt (line 12)) (2.8)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests~=2.24.0->paddlefsl==1.0.0->paddlenlp->-r requirements.txt (line 12)) (1.25.6)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests~=2.24.0->paddlefsl==1.0.0->paddlenlp->-r requirements.txt (line 12)) (2019.9.11)
Requirement already satisfied: Flask-Babel>=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (1.0.0)
Requirement already satisfied: bce-python-sdk in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (0.8.53)
Requirement already satisfied: flask>=1.1.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (1.1.1)
Requirement already satisfied: pre-commit in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (1.21.0)
Requirement already satisfied: protobuf>=3.11.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (3.14.0)
Requirement already satisfied: shellcheck-py in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (0.7.1.1)
Requirement already satisfied: importlib-metadata<4.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8->motmetrics->-r requirements.txt (line 19)) (4.2.0)
Requirement already satisfied: pycodestyle<2.9.0,>=2.8.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8->motmetrics->-r requirements.txt (line 19)) (2.8.0)
Requirement already satisfied: pyflakes<2.5.0,>=2.4.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8->motmetrics->-r requirements.txt (line 19)) (2.4.0)
Requirement already satisfied: mccabe<0.7.0,>=0.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flake8->motmetrics->-r requirements.txt (line 19)) (0.6.1)
Requirement already satisfied: dill>=0.3.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from multiprocess->paddlenlp->-r requirements.txt (line 12)) (0.3.3)
Collecting iniconfig
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/9b/dd/b3c12c6d707058fa947864b67f0c4e0c39ef8610988d7baea9578f3c48f3/iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB)
Requirement already satisfied: attrs>=19.2.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pytest->motmetrics->-r requirements.txt (line 19)) (21.4.0)
Requirement already satisfied: pluggy<2.0,>=0.12 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pytest->motmetrics->-r requirements.txt (line 19)) (0.13.1)
Collecting py>=1.8.2
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/f6/f0/10642828a8dfb741e5f3fbaac830550a518a775c7fff6f04a007259b0548/py-1.11.0-py2.py3-none-any.whl (98 kB)
     |████████████████████████████████| 98 kB 402 kB/s             
[?25hCollecting tomli>=1.0.0
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl (12 kB)
Collecting py-cpuinfo
  Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e6/ba/77120e44cbe9719152415b97d5bfb29f4053ee987d6cb63f55ce7d50fadc/py-cpuinfo-8.0.0.tar.gz (99 kB)
     |████████████████████████████████| 99 kB 2.3 MB/s             
[?25h  Preparing metadata (setup.py) ... [?25ldone
[?25hRequirement already satisfied: scikit-learn>=0.21.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from seqeval->paddlenlp->-r requirements.txt (line 12)) (0.24.2)
Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (1.1.0)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (0.16.0)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (2.11.0)
Requirement already satisfied: click>=5.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (7.0)
Requirement already satisfied: Babel>=2.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (2.8.0)
Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from importlib-metadata<4.3->flake8->motmetrics->-r requirements.txt (line 19)) (3.7.0)
Requirement already satisfied: typing-extensions>=3.6.4 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from importlib-metadata<4.3->flake8->motmetrics->-r requirements.txt (line 19)) (4.0.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp->-r requirements.txt (line 12)) (2.1.0)
Requirement already satisfied: joblib>=0.11 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp->-r requirements.txt (line 12)) (0.14.1)
Requirement already satisfied: pycryptodome>=3.8.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from bce-python-sdk->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (3.9.9)
Requirement already satisfied: virtualenv>=15.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (16.7.9)
Requirement already satisfied: nodeenv>=0.11.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (1.3.4)
Requirement already satisfied: identify>=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (1.4.10)
Requirement already satisfied: toml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (0.10.0)
Requirement already satisfied: aspy.yaml in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (1.3.0)
Requirement already satisfied: cfgv>=2.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from pre-commit->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (2.0.1)
Requirement already satisfied: MarkupSafe>=0.23 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask>=1.1.1->visualdl>=2.1.0->paddledet->-r requirements.txt (line 16)) (2.0.1)
Building wheels for collected packages: lap, cython-bbox, pycocotools, py-cpuinfo
  Building wheel for lap (setup.py) ... [?25ldone
[?25h  Created wheel for lap: filename=lap-0.4.0-cp37-cp37m-linux_x86_64.whl size=1593870 sha256=383f558707009f930cf7d69e2b4a3ae1ad2c8b6f1d80d7c7199b6d11efc96a69
  Stored in directory: /home/aistudio/.cache/pip/wheels/5c/d0/d2/e331d17a999666b1e2eb99743cfa1742629f9d26c55c657001
  Building wheel for cython-bbox (setup.py) ... [?25ldone
[?25h  Created wheel for cython-bbox: filename=cython_bbox-0.1.3-cp37-cp37m-linux_x86_64.whl size=61619 sha256=fee7bccbe8214d10185a65aac357bff45ad92e0eca525b7688617b6eb2dd4e3f
  Stored in directory: /home/aistudio/.cache/pip/wheels/3e/b3/6a/aae8832326545645e65d643a2aaf223ffa3a7d01e1a1bae01b
  Building wheel for pycocotools (pyproject.toml) ... [?25ldone
[?25h  Created wheel for pycocotools: filename=pycocotools-2.0.4-cp37-cp37m-linux_x86_64.whl size=273789 sha256=c4019e01fa4dea259cf6072dd3b17f524a64d0955bf1cb09f94bc11504113fa7
  Stored in directory: /home/aistudio/.cache/pip/wheels/c0/01/5f/670dfd20204fc9cc6bf843db4e014acb998f411922e3abc49f
  Building wheel for py-cpuinfo (setup.py) ... [?25ldone
[?25h  Created wheel for py-cpuinfo: filename=py_cpuinfo-8.0.0-py3-none-any.whl size=22245 sha256=7a5755a906124d042677008953be77620053fd5ee6e03c0b043118fe42273e62
  Stored in directory: /home/aistudio/.cache/pip/wheels/88/c7/d0/6309c7cc9929894c11fe8e516c3e2a0d0a53ee4e198eac48b7
Successfully built lap cython-bbox pycocotools py-cpuinfo
Installing collected packages: tomli, py, iniconfig, pytest, py-cpuinfo, xmltodict, pytest-benchmark, flake8-import-order, typeguard, tifffile, terminaltables, shapely, PyWavelets, pycocotools, opencv-python, motmetrics, lap, cython-bbox, SimpleITK, scikit-image, paddledet, lmdb, ffmpeg-python, decord, av
  Attempting uninstall: opencv-python
    Found existing installation: opencv-python 4.1.1.26
    Uninstalling opencv-python-4.1.1.26:
      Successfully uninstalled opencv-python-4.1.1.26
Successfully installed PyWavelets-1.2.0 SimpleITK-2.1.1 av-8.0.3 cython-bbox-0.1.3 decord-0.4.2 ffmpeg-python-0.2.0 flake8-import-order-0.18.1 iniconfig-1.1.1 lap-0.4.0 lmdb-1.3.0 motmetrics-1.2.0 opencv-python-4.2.0.32 paddledet-2.3.0 py-1.11.0 py-cpuinfo-8.0.0 pycocotools-2.0.4 pytest-7.0.1 pytest-benchmark-3.4.1 scikit-image-0.19.2 shapely-1.8.1.post1 terminaltables-3.1.10 tifffile-2021.11.2 tomli-2.0.1 typeguard-2.13.3 xmltodict-0.12.0
[33mWARNING: You are using pip version 21.3.1; however, version 22.0.3 is available.
You should consider upgrading via the '/opt/conda/envs/python35-paddle120-env/bin/python -m pip install --upgrade pip' command.[0m

抽帧时间较长，大约需要执行10分钟。

%cd /home/aistudio/PaddleVideo/applications/FootballAction
!python3.7 datasets/script/get_frames_pcm.py

/home/aistudio/PaddleVideo/applications/FootballAction
mp4/football.mp4
done

数据预处理后得到的文件夹格式如下：

|--  datasets                   # 训练数据集和处理脚本
        |--  EuroCup2016   # 数据集
            |--  mp4               # 原始视频.mp4
            |--  frames           # 图像帧
            |--  pcm               # 音频pcm
            |--  url.list            # 视频列表
            |--  label.json       # 视频原始gts

5. 模型训练

5.1 PP-TSM训练

5.1.1 PP-TSM 训练数据处理

PP-TSM模型是一个视频理解模型，它可以用于包含单动作的视频段的视频分类任务。在本方案中，我们使用训练好的PP-TSM模型提取视频图像特征。

get_instance_for_pptsm.py 文件用于生成训练所需的正负样本。正样本为标注后的运动区间，该区间内的所有图像帧会生成一个pkl文件；负样本为标注后的非运动区间，因为足球赛事中无特殊动作的时间较长，负样本以随机取N个区间生成N个pkl的方式生成。

!python datasets/script/get_instance_for_pptsm.py

完成该步骤后，数据存储位置如下：

|--  datasets                   # 训练数据集和处理脚本
        |--  EuroCup2016            # 数据集
            |--  input_for_pptsm   # pptsm训练的数据

5.1.2 PP-TSM模型训练

在训练开始前，需要先下载图像蒸馏预训练模型ResNet50_vd_ssld_v2.pdparams 作为模型 backbone 初始化参数，通过如下命令下载：

# 在第一次运行代码时执行如下命令
!wget https://videotag.bj.bcebos.com/PaddleVideo/PretrainModel/ResNet50_vd_ssld_v2_pretrained.pdparams
!mkdir pretrain
!mv ResNet50_vd_ssld_v2_pretrained.pdparams pretrain/ResNet50_vd_ssld_v2_pretrained.pdparams

启动训练，可以在 pptsm_football_v2.0.yaml 中调节训练参数。

# 启动训练
%cd /home/aistudio/PaddleVideo/
!python -B -m paddle.distributed.launch \
    --gpus="0" \
    --log_dir=applications/FootballAction/train_pptsm/logs \
    main.py \
    --validate \
    -c applications/FootballAction/train_proposal/configs/pptsm_football_v2.0.yaml  \
    -o output_dir=applications/FootballAction/train_pptsm/

5.1.3 PP-TSM模型转为预测模式

使用如下命令生成预测所需的模型结构文件ppTSM.pdmodel和模型权重文件ppTSM.pdiparams。

!python tools/export_model.py -c applications/FootballAction/train_proposal/configs/pptsm_football_v2.0.yaml \
                               -p applications/FootballAction/train_pptsm/ppTSM_best.pdparams \
                               -o applications/FootballAction/checkpoints/ppTSM

5.1.4 基于PP-TSM的视频特征提取

基于刚训练好的PP-TSM模型提取视频特征。每次输入8帧图像，输出8维特征，最后将所有特征拼接在一起代表视频特征。

extract_feat.py 文件同时进行视频图像特征和音频特征提取，音频特征通过VGGish模型提取。该模型权重文件可通过如下命令下载：

# 仅在第一次运行notebook时下载
!wget https://videotag.bj.bcebos.com/PaddleVideo-release2.1/FootballAction/audio.tar
!tar -xvf audio.tar
!rm audio.tar
!mv AUDIO/ /home/aistudio/PaddleVideo/applications/FootballAction/checkpoints/

提取视频图像和音频特征：

%cd /home/aistudio/PaddleVideo/applications/FootballAction/extractor
!python extract_feat.py

特征提取完成后存储在EuroCup2016文件夹下

   |--  datasets                   # 训练数据集和处理脚本
        |--  EuroCup2016     # 数据集
            |--  features           # 视频的图像+音频特征

5.2 BMN训练

将得到的音视频特征整合输入BMN网络，得到由动作开始时间和结束时间组合成的时序片段（proposal）。

5.2.1 BMN训练数据处理

get_instance_for_bmn 文件用于提取二分类的proposal，根据标注文件和音视频特征得到BMN所需要的数据集。

%cd /home/aistudio/PaddleVideo/applications/FootballAction
!python datasets/script/get_instance_for_bmn.py

得到的数据格式如下：

duration_second 代表视频片段时长，duration_frame 代表涵盖多少帧，annotations为这段视频的真实标注，segment为动作起止时间，label及label_name 为动作类别和名称。

{
    "719b0a4bcb1f461eabb152298406b861_753_793": {
        "duration_second": 40.0,
        "duration_frame": 200,
        "feature_frame": 200,
        "subset": "train",
        "annotations": [
            {
                "segment": [
                    15.0,
                    22.0
                ],
                "label": "3.0",
                "label_name": "任意球"
            }
        ]
    },
    ...
}

完成该步骤后，数据存储位置如下：

   |--  datasets                   # 训练数据集和处理脚本
        |--  EuroCup2016            # xx数据集
            |--  input_for_bmn     # bmn训练的proposal

5.2.2 BMN模型训练

该步骤训练与PP-TSM模型训练类似，可以调整 bmn_football_v2.0.yaml 文件，修改训练参数。

%cd /home/aistudio/PaddleVideo/
!python -B -m paddle.distributed.launch \
    --gpus='0' \
    --log_dir=applications/FootballAction/train_bmn/logs \
    main.py \
    --validate \
    -c applications/FootballAction/train_proposal/configs/bmn_football_v2.0.yaml \
    -o output_dir=applications/FootballAction/train_bmn/

5.2.3 BMN模型转为预测模式

生成预测所需的模型结构文件BMN.pdmodel和模型权重文件BMN.pdiparams。

!python tools/export_model.py \
    -c applications/FootballAction/train_proposal/configs/bmn_football_v2.0.yaml \
    -p applications/FootballAction/train_bmn/BMN_epoch_00001.pdparams \
    -o applications/FootballAction/checkpoints/BMN

5.2.4 BMN模型预测

进行模型预测，得到动作proposal的信息，包括动作开始时间、结束时间以及置信度。

%cd /home/aistudio/PaddleVideo/applications/FootballAction/extractor
!python extract_bmn.py

完成该步骤后，数据存储位置

   |--  datasets                   # 训练数据集和处理脚本
        |--  EuroCup2016            # xx数据集
            |--  feature_bmn
                 |--  prop.json    # bmn 预测结果

5.3 LSTM训练

得到时序片段提名（proposal）后，根据动作开始和结束时间阶段视频和音频特征，通过AttentionLSTM输出动作类别。

5.3.1 LSTM训练数据处理

按照BMN预测到的proposal截断视频特征，生成训练AttentionLSTM所需数据集。

%cd /home/aistudio/PaddleVideo/applications/FootballAction/datasets/script/
!python get_instance_for_lstm.py

5.3.2 LSTM训练

通过如下命令训练AttentionLSTM网络。

cd /home/aistudio/PaddleVideo/applications/FootballAction/train_lstm/
python -u scenario_lib/train.py  \
    --model_name=ActionNet \
    --config=conf/conf.txt \
    --save_dir=../football_lstm

这里因为训练数据只提供了一条视频，直接进行LSTM训练会存在问题，如有需要可根据自己的数据集使用上述命令进行训练，并通过修改 conf/conf.txt 文件，更改模型参数。

5.3.3 LSTM模型转为预测模式

如需将训练模型转为预测模式，可通过以下命令生成模型结构文件。

cd /home/aistudio/PaddleVideo/applications/FootballAction/train_lstm/

python inference_model.py --config=conf/conf.yaml --weights=../football_lstm/ActionNet.pdparams --save_dir=../checkpoints/LSTM

我们提供训练好的LSTM模型，共大家进行后续测试。

!wget https://videotag.bj.bcebos.com/PaddleVideo-release2.1/FootballAction/lstm.tar
!tar -xvf lstm.tar
!rm lstm.tar
!mv LSTM/ /home/aistudio/PaddleVideo/applications/FootballAction/checkpoints/

6. 模型推理

输入一条视频数据，以该网络结构进行推理。

%cd /home/aistudio/PaddleVideo/applications/FootballAction/predict
!python predict.py

7. 模型评估

通过如下命令开始模型评估。

# 包括bmn proposal 评估和最终action评估
%cd /home/aistudio/PaddleVideo/applications/FootballAction/predict
!python eval.py results.json

8. 模型优化

本案例中展示的PP-TSM+BMN+AttentionLSTM的实现方案是经过多次优化实验得来的。在实验的初始阶段，我们最先选取的是TSN+BMN+AttentionLSTM的实现方案。后续经过三个方面的大量优化处理，才在模型效果提升方面有了显著成果。可以通过阅读本节获得在模型效果提升方面的宝贵经验。

实验分为以下三个方面：

  • 更换图像特征提取模型，分别采用视频理解模型TSN、TSM、PP-TSM 进行实验，在动作识别任务上精度更高的模型，在动作检测任务上也有更好的表现。
  • 在BMN生成的时序提名基础上扩展proposal特征，更好的涵盖整个动作特征
  • 增加训练数据量，使网络具有更好的泛化能力

实验最初采用TSN+BMN+AttentionLSTM的解决方案。实验Baseline如下：

图像特征提取模型top1 acc时序提名生成序列分类precisionrecallF1-score
TSN75.86BMNLSTM60.0461.3160.67

通过将TSN更换为在视频理解任务上精度更好的TSM模型得到如下效果提升：

图像特征提取模型top1 acc时序提名生成序列分类precisionrecallF1-score
TSN75.86BMNLSTM60.0461.3160.67
TSM90.24BMNLSTM71.0665.9368.4

可以看到，TSN和TSM模型在视频分类任务上的精度分别是75.86%和90.24%，精度大幅提升，使得提取的视频图像特征更为准确，最终在行为检测任务上F1-score由60.67%提升到68.4%。

可以通过如下代码训练TSN、TSM网络，进行数据对比。

# 训练TSN
%cd /home/aistudio/PaddleVideo/
!python -B -m paddle.distributed.launch \
    --gpus="0" \
    --log_dir=applications/FootballAction/train_tsn/logs \
    main.py \
    --validate \
    -c applications/FootballAction/train_proposal/configs/tsn_football.yaml  \
    -o output_dir=applications/FootballAction/train_tsn/

# 训练TSM
%cd /home/aistudio/PaddleVideo/
!python -B -m paddle.distributed.launch \
    --gpus="0" \
    --log_dir=applications/FootballAction/train_tsm/logs \
    main.py \
    --validate \
    -c applications/FootballAction/train_proposal/configs/tsm_football.yaml  \
    -o output_dir=applications/FootballAction/train_tsm/

除此之外，我们还查验了BMN生成的proposals，发现某些时间片段并不能完整涵盖动作。为更好的涵盖整个动作特征，我们将生成的proposal前后各扩展1s。

实验对比数据如下：

图像特征提取模型top1 acc时序提名生成序列分类precisionrecallF1-score
TSM90.24BMNLSTM71.0665.9368.4
TSM90.24BMNLSTM + 扩展全部动作的proposal80.366.3172.64

对所有动作扩展proposal后我们发现，并不是所有动作扩展proposal特征都有明显的正向提升，因此我们选择只对部分特征扩展proposal，得到如下实验结果：

图像特征提取模型top1 acc时序提名生成序列分类precisionrecallF1-score
TSM90.24BMNLSTM + 扩展全部动作的proposal80.366.3172.64
TSM90.24BMNLSTM + 扩展部分动作的proposal79.3767.7673.1

可通过以下代码扩展proposal特征：

%cd /home/aistudio/PaddleVideo/applications/FootballAction/datasets/script/
!python get_instance_for_lstm_long_proposal.py

以上所有实验均基于由156个训练集+15个测试集组成的数据集，经过一轮新的数据标注，我们将数据集扩充为272训练集+25测试集，并基于此进行了新的训练，得到如下结果：

图像特征提取模型top1 acc时序提名生成序列分类precisionrecallF1-score
TSM90.24BMNLSTM + 扩展部分动作的proposal79.3767.7673.1
TSM + 数据扩展90.85BMNLSTM + 扩展部分动作的proposal80.2268.7274.03

最后，我们将TSM模型更换为精度更高的PP-TSM模型，得到最终的实验结果。

图像特征提取模型top1 acc时序提名生成序列分类precisionrecallF1-score
TSM + 数据扩展90.85BMNLSTM + 扩展部分动作的proposal80.2268.7274.03
PP-TSM + 数据扩展91.03BMNLSTM + 扩展部分动作的proposal81.271.876.2

以上所有实验的对比数据如下：

图像特征提取模型top1 acc时序提名生成序列分类precisionrecallF1-score
TSN75.86BMNLSTM60.0461.3160.67
TSM90.24BMNLSTM71.0665.9368.4
TSM90.24BMNLSTM + 扩展全部动作的proposal80.366.3172.64
TSM90.24BMNLSTM + 扩展部分动作的proposal79.3767.7673.1
TSM + 数据扩展90.85BMNLSTM + 扩展部分动作的proposal80.2268.7274.03
PP-TSM + 数据扩展91.03BMNLSTM + 扩展部分动作的proposal81.271.876.2

欢迎报名直播课加入交流群，如需更多技术交流与合作可点击以下链接
https://paddleqiyeban.wjx.cn/vj/Qlb0uS3.aspx?udsid=531417

参考文献

  • Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, Shilei Wen,2019. BMN: Boundary-Matching Network for Temporal Action Proposal Generation. https://arxiv.org/pdf/1907.09702.pdf

资源

更多资源请参考：

# lstm # 人工智能 # 深度学习 # paddlepaddle
Logo
百度飞桨AI Studio社区

学大模型，用大模型上飞桨星河社区！每天8点V100G算力免费领！免费领取ERNIE 4.0 100w Token >>>

更多推荐

cover

利用Amazon Bedrock畅玩Claude 3等多种领先模型，抢占AI高地(体验倒计时4小时)

avatar 百度飞桨星河社区
cover

RAPTOR：索引树状 RAG，使用树结构来捕捉文本的高级和低级细节

avatar 百度飞桨星河社区

RAG介绍

LlamaIndex是一个基于大型语言模型（LLM）的应用程序数据框架，旨在连接LLM与外部数据，特别是私有或特定领域的数据，以实现更准确的文本生成和知识增强。

avatar 百度飞桨星河社区