【PaddleNLP实战】对话意图分类方案
意图分类是自然语言理解(NLU)中的一项重要任务，在智能客服、对话机器人、搜索引擎等各个领域有重要作用。
·
基于预训练模型ERNIE 3.0 和CrossWOZ数据的意图识别分类任务
AI Studio平台默认安装了Paddle和PaddleNLP，并定期更新版本。 如需手动更新Paddle，可参考飞桨安装说明，安装相应环境下最新版飞桨框架。
使用如下命令确保安装最新版PaddleNLP：
# 首次更新完以后，重启后方能生效
!pip install --upgrade paddlenlp
Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple
Requirement already satisfied: paddlenlp in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (2.1.1)
Collecting paddlenlp
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/1f/54/6d5ae82c9a4556daf18334a07d8dd4612b586b4bd4629bcd82099cccf20d/paddlenlp-2.4.1-py3-none-any.whl (1.9 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m1.9/1.9 MB[0m [31m1.3 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m0m
[?25hRequirement already satisfied: tqdm in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (4.27.0)
Requirement already satisfied: sentencepiece in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (0.1.96)
Requirement already satisfied: paddlefsl in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (1.0.0)
Requirement already satisfied: jieba in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (0.42.1)
Requirement already satisfied: dill<0.3.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (0.3.3)
Collecting datasets>=2.0.0
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7c/75/d282907e7ebd87e4b3475bc5156140465372fa451bc6cbddbefa54915d00/datasets-2.6.1-py3-none-any.whl (441 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m441.9/441.9 kB[0m [31m2.4 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hRequirement already satisfied: protobuf<=3.20.0,>=3.1.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (3.20.0)
Requirement already satisfied: colorlog in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (4.1.0)
Requirement already satisfied: colorama in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (0.4.4)
Requirement already satisfied: seqeval in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (1.2.2)
Requirement already satisfied: paddle2onnx in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (1.0.0)
Requirement already satisfied: visualdl in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (2.4.0)
Requirement already satisfied: multiprocess<=0.70.12.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from paddlenlp) (0.70.11.1)
Collecting huggingface-hub<1.0.0,>=0.2.0
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/b2/2b/715a1924d470691a27b2dcdf472a9ef87f04718a897de25e68bf86ac0184/huggingface_hub-0.10.1-py3-none-any.whl (163 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m163.5/163.5 kB[0m [31m99.1 kB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hCollecting aiohttp
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7a/48/7882af39221fee58e33eee6c8e516097e2331334a5937f54fe5b5b285d9e/aiohttp-3.8.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (948 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m948.0/948.0 kB[0m [31m3.2 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hRequirement already satisfied: importlib-metadata in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from datasets>=2.0.0->paddlenlp) (4.2.0)
Requirement already satisfied: pandas in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from datasets>=2.0.0->paddlenlp) (1.1.5)
Requirement already satisfied: packaging in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from datasets>=2.0.0->paddlenlp) (21.3)
Requirement already satisfied: numpy>=1.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from datasets>=2.0.0->paddlenlp) (1.19.5)
Collecting tqdm
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/47/bb/849011636c4da2e44f1253cd927cfb20ada4374d8b3a4e425416e84900cc/tqdm-4.64.1-py2.py3-none-any.whl (78 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m78.5/78.5 kB[0m [31m3.1 MB/s[0m eta [36m0:00:00[0m
[?25hCollecting responses<0.19
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/79/f3/2b3a6dc5986303b3dd1bbbcf482022acb2583c428cd23f0b6d37b1a1a519/responses-0.18.0-py3-none-any.whl (38 kB)
Requirement already satisfied: pyyaml>=5.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from datasets>=2.0.0->paddlenlp) (5.1.2)
Collecting fsspec[http]>=2021.11.1
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/82/f3/30f7925f22f623ebac35b40d48151578ef7303d897764e1d95323727611b/fsspec-2022.8.2-py3-none-any.whl (140 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m140.8/140.8 kB[0m [31m3.3 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hCollecting xxhash
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/ef/ac/0eb796cf052c392f1ae586452c89ca66164f7d4f655b039ca3d06e2291af/xxhash-3.0.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (212 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m212.2/212.2 kB[0m [31m1.8 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hRequirement already satisfied: requests>=2.19.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from datasets>=2.0.0->paddlenlp) (2.24.0)
Collecting pyarrow>=6.0.0
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/7f/08/9b5fe7c9e2774bca77dae29d22a446ead804fb8e050f2899ae1f60d73ad1/pyarrow-9.0.0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (35.3 MB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m35.3/35.3 MB[0m [31m1.8 MB/s[0m eta [36m0:00:00[0m00:01[0m00:01[0m
[?25hCollecting paddlefsl
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/fb/4a/25d1959a8f1fe5ee400f32fc9fc8b56d4fd6fc25315e23c0171f6e705e2a/paddlefsl-1.1.0-py3-none-any.whl (101 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m101.0/101.0 kB[0m [31m135.1 kB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hRequirement already satisfied: scikit-learn>=0.21.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from seqeval->paddlenlp) (0.24.2)
Requirement already satisfied: Pillow>=7.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (8.2.0)
Requirement already satisfied: Flask-Babel>=1.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (1.0.0)
Requirement already satisfied: bce-python-sdk in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (0.8.53)
Requirement already satisfied: flask>=1.1.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (1.1.1)
Requirement already satisfied: matplotlib in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (2.2.3)
Requirement already satisfied: six>=1.14.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from visualdl->paddlenlp) (1.16.0)
Requirement already satisfied: itsdangerous>=0.24 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->paddlenlp) (1.1.0)
Requirement already satisfied: Jinja2>=2.10.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->paddlenlp) (3.0.0)
Requirement already satisfied: Werkzeug>=0.15 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->paddlenlp) (0.16.0)
Requirement already satisfied: click>=5.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from flask>=1.1.1->visualdl->paddlenlp) (7.0)
Requirement already satisfied: Babel>=2.3 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl->paddlenlp) (2.8.0)
Requirement already satisfied: pytz in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Flask-Babel>=1.0.0->visualdl->paddlenlp) (2019.3)
Collecting frozenlist>=1.1.1
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/3e/b2/cf7e86583f03fafc93c4103f9a03aaf729dcf4dca9cd3012256a48b766ad/frozenlist-1.3.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (148 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m148.0/148.0 kB[0m [31m1.9 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hRequirement already satisfied: typing-extensions>=3.7.4 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from aiohttp->datasets>=2.0.0->paddlenlp) (4.3.0)
Collecting yarl<2.0,>=1.0
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/2b/89/36a50cab1be3d5099ec66a41212cf0c11507c343074e97e907a2f5f1a569/yarl-1.8.1-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (231 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m231.3/231.3 kB[0m [31m2.1 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hRequirement already satisfied: attrs>=17.3.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from aiohttp->datasets>=2.0.0->paddlenlp) (22.1.0)
Collecting async-timeout<5.0,>=4.0.0a3
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/d6/c1/8991e7c5385b897b8c020cdaad718c5b087a6626d1d11a23e1ea87e325a7/async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
Collecting charset-normalizer<3.0,>=2.0
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/db/51/a507c856293ab05cdc1db77ff4bc1268ddd39f29e7dc4919aa497f0adbec/charset_normalizer-2.1.1-py3-none-any.whl (39 kB)
Collecting asynctest==0.13.0
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/e8/b6/8d17e169d577ca7678b11cd0d3ceebb0a6089a7f4a2de4b945fe4b1c86db/asynctest-0.13.0-py3-none-any.whl (26 kB)
Collecting multidict<7.0,>=4.5
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/db/3f/1c876ed190e8fcd1a2faef3085427e5465076e28813a2499502633f7eed3/multidict-6.0.2-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (94 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m94.8/94.8 kB[0m [31m324.3 kB/s[0m eta [36m0:00:00[0m [36m0:00:01[0m
[?25hCollecting aiosignal>=1.1.2
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/3b/87/fe94898f2d44a93a35d5aa74671ed28094d80753a1113d68b799fab6dc22/aiosignal-1.2.0-py3-none-any.whl (8.2 kB)
Requirement already satisfied: filelock in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets>=2.0.0->paddlenlp) (3.0.12)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from packaging->datasets>=2.0.0->paddlenlp) (3.0.9)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.19.0->datasets>=2.0.0->paddlenlp) (2.8)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.19.0->datasets>=2.0.0->paddlenlp) (2019.9.11)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.19.0->datasets>=2.0.0->paddlenlp) (1.25.6)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from requests>=2.19.0->datasets>=2.0.0->paddlenlp) (3.0.4)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1
Downloading https://pypi.tuna.tsinghua.edu.cn/packages/56/aa/4ef5aa67a9a62505db124a5cb5262332d1d4153462eb8fd89c9fa41e5d92/urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
[2K [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m128.0/128.0 kB[0m [31m1.5 MB/s[0m eta [36m0:00:00[0ma [36m0:00:01[0m
[?25hRequirement already satisfied: joblib>=0.11 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp) (0.14.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp) (2.1.0)
Requirement already satisfied: scipy>=0.19.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from scikit-learn>=0.21.3->seqeval->paddlenlp) (1.6.3)
Requirement already satisfied: future>=0.6.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from bce-python-sdk->visualdl->paddlenlp) (0.18.0)
Requirement already satisfied: pycryptodome>=3.8.0 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from bce-python-sdk->visualdl->paddlenlp) (3.9.9)
Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from importlib-metadata->datasets>=2.0.0->paddlenlp) (3.8.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->visualdl->paddlenlp) (1.1.0)
Requirement already satisfied: cycler>=0.10 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->visualdl->paddlenlp) (0.10.0)
Requirement already satisfied: python-dateutil>=2.1 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from matplotlib->visualdl->paddlenlp) (2.8.2)
Requirement already satisfied: MarkupSafe>=2.0.0rc2 in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from Jinja2>=2.10.1->flask>=1.1.1->visualdl->paddlenlp) (2.0.1)
Requirement already satisfied: setuptools in /opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages (from kiwisolver>=1.0.1->matplotlib->visualdl->paddlenlp) (56.2.0)
Installing collected packages: xxhash, urllib3, tqdm, pyarrow, multidict, fsspec, frozenlist, charset-normalizer, asynctest, async-timeout, yarl, aiosignal, responses, paddlefsl, huggingface-hub, aiohttp, datasets, paddlenlp
Attempting uninstall: urllib3
Found existing installation: urllib3 1.25.6
Uninstalling urllib3-1.25.6:
Successfully uninstalled urllib3-1.25.6
Attempting uninstall: tqdm
Found existing installation: tqdm 4.27.0
Uninstalling tqdm-4.27.0:
Successfully uninstalled tqdm-4.27.0
Attempting uninstall: pyarrow
Found existing installation: pyarrow 2.0.0
Uninstalling pyarrow-2.0.0:
Successfully uninstalled pyarrow-2.0.0
Attempting uninstall: paddlefsl
Found existing installation: paddlefsl 1.0.0
Uninstalling paddlefsl-1.0.0:
Successfully uninstalled paddlefsl-1.0.0
Attempting uninstall: paddlenlp
Found existing installation: paddlenlp 2.1.1
Uninstalling paddlenlp-2.1.1:
Successfully uninstalled paddlenlp-2.1.1
[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
parl 1.4.1 requires pyzmq==18.1.1, but you have pyzmq 23.2.1 which is incompatible.[0m[31m
[0mSuccessfully installed aiohttp-3.8.3 aiosignal-1.2.0 async-timeout-4.0.2 asynctest-0.13.0 charset-normalizer-2.1.1 datasets-2.6.1 frozenlist-1.3.1 fsspec-2022.8.2 huggingface-hub-0.10.1 multidict-6.0.2 paddlefsl-1.1.0 paddlenlp-2.4.1 pyarrow-9.0.0 responses-0.18.0 tqdm-4.64.1 urllib3-1.25.11 xxhash-3.0.0 yarl-1.8.1
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m A new release of pip available: [0m[31;49m22.1.2[0m[39;49m -> [0m[32;49m22.3[0m
[1m[[0m[34;49mnotice[0m[1;39;49m][0m[39;49m To update, run: [0m[32;49mpip install --upgrade pip[0m
1.任务目标
意图识别是指分析用户的核心需求，输出与查询输入最相关的信息。在对话过程中要准确理解对方所想表达的意思，这是具有很大挑战性的任务。 例如用户输入查询“仙剑奇侠传”时，我们知道“仙剑奇侠传”既有游戏又有电视剧还有新闻、图片等等，如果我们通过用户意图识别发现该用户是想看“仙剑奇侠传”电视剧的，那我们直接把电视剧作为结果返回给用户，就会节省用户的搜索点击次数提升使用体验。而在对话中如果对方说“我的苹果从不出现卡顿”，那么我们就能通过意图识别判断出此刻的苹果是一个电子设备，而非水果，这样对话就能顺利进行下去。总之，意图识别的准确性能够提高搜索的匹配度和对话系统的智能性。
我们可以将意图识别视为一个文本分类问题，使用自然语言处理方法解决该问题，本节将以CrossWOZ数据集抽取的中文对话意图识别分类为例，介绍飞桨平台上实现基于预训练模型意图识别多标签分类具体过程。
2.实现思路及流程
基于预训练模型实现思路如下图所示，模型的输入是文本数据，模型输出是文本的意图分类标签。在建模过程中，对输入的对话文本，首先进行数据处理，使用预训练模型进行文本序列编码，获得文本的语义向量表示，最后经过全连接层和Sigmoid处理得到文本在每个意图分析类别标签的概率。
根据上面的介绍，基于飞桨框架实现意图识别的实现过程包含如下6个步骤：
（1）模型构建：确定意图识别分类任务使用的模型，本实践使用ERNIE 3.0 Medium模型获取文本序列的向量表示。
（2）数据处理：对于输入的文本数据进行相应的处理，包括词语切分、文本截取和填充、词到词号的转化等，然后数据组成minibatch格式传入到预训练模型中。
（3）训练配置：实例化模型，并选择模型的计算资源（CPU或者GPU），指定模型迭代的优化算法。
（4）模型训练：训练模型的参数，以达到最优效果。
（5）模型评估：对训练好的模型进行评估测试，观察准确率和损失函数的变化情况。
（6）模型预测：选取一段对话文本，通过模型预测相应意图类别。
以下分别介绍每个步骤的具体实现过程。
3. 模型构建
我们使用ERNIE 3.0 Medium作为预训练模型用于意图分析分类。可以直接使用PaddleNLP中Auto模块（包括AutoModel, AutoTokenizer及各种下游任务类）调用不同网络结构的预训练模型和对应的分词器。AutoModelForSequenceClassification可用于文本分类，通过预训练模型获取输入文本的表示，之后将文本表示进行分类。
ERNIE 3.0首次在百亿级预训练模型中引入大规模知识图谱，提出了海量无监督文本与大规模知识图谱的平行预训练方法(Universal Knowledge-Text Prediction)，通过将知识图谱挖掘算法得到五千万知识图谱三元组与4TB大规模语料同时输入到预训练模型中进行联合掩码训练，促进了结构化知识和无结构文本之间的信息共享，大幅提升了模型对于知识的记忆和推理能力。
ERNIE 3.0框架分为两层。第一层是通用语义表示网络，该网络学习数据中的基础和通用的知识。第二层是任务语义表示网络，该网络基于通用语义表示，学习任务相关的知识。在学习过程中，任务语义表示网络只学习对应类别的预训练任务，而通用语义表示网络会学习所有的预训练任务。
# 使用Auto模块加载ERNIE 3.0 Medium预训练模型和分词器
from paddlenlp.transformers import AutoModelForSequenceClassification, AutoTokenizer
model = AutoModelForSequenceClassification.from_pretrained("ernie-3.0-medium-zh", num_classes=158)
tokenizer = AutoTokenizer.from_pretrained("ernie-3.0-medium-zh")
[2022-10-19 12:38:03,670] [ INFO] - We are using <class 'paddlenlp.transformers.ernie.modeling.ErnieForSequenceClassification'> to load 'ernie-3.0-medium-zh'.
[2022-10-19 12:38:03,673] [ INFO] - Downloading https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh.pdparams and saved to /home/aistudio/.paddlenlp/models/ernie-3.0-medium-zh
[2022-10-19 12:38:03,675] [ INFO] - Downloading ernie_3.0_medium_zh.pdparams from https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh.pdparams
100%|██████████| 313M/313M [00:13<00:00, 25.0MB/s]
W1019 12:38:16.869254 98 gpu_resources.cc:61] Please NOTE: device: 0, GPU Compute Capability: 8.0, Driver API Version: 11.2, Runtime API Version: 11.2
W1019 12:38:16.874008 98 gpu_resources.cc:91] device: 0, cuDNN Version: 8.2.
[2022-10-19 12:38:19,584] [ INFO] - We are using <class 'paddlenlp.transformers.ernie.tokenizer.ErnieTokenizer'> to load 'ernie-3.0-medium-zh'.
[2022-10-19 12:38:19,587] [ INFO] - Downloading https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh_vocab.txt and saved to /home/aistudio/.paddlenlp/models/ernie-3.0-medium-zh
[2022-10-19 12:38:19,589] [ INFO] - Downloading ernie_3.0_medium_zh_vocab.txt from https://bj.bcebos.com/paddlenlp/models/transformers/ernie_3.0/ernie_3.0_medium_zh_vocab.txt
100%|██████████| 182k/182k [00:00<00:00, 1.45MB/s]
[2022-10-19 12:38:19,867] [ INFO] - tokenizer config file saved in /home/aistudio/.paddlenlp/models/ernie-3.0-medium-zh/tokenizer_config.json
[2022-10-19 12:38:19,869] [ INFO] - Special tokens file saved in /home/aistudio/.paddlenlp/models/ernie-3.0-medium-zh/special_tokens_map.json
4.数据处理
数据处理包括数据集确定、CrossWOZ数据集加载、数据预处理、构造DataLoader等步骤。
（1）数据集确定
CrossWOZ是一个中文大规模跨领域任务导向对话数据（Cross-Domain Wizard-of-Oz task-oriented）集包含 6K 个对话，102K 个句子，涉及 5 个领域（景点、酒店、餐馆、地铁、出租）。平均每个对话涉及 3.2 个领域，远超之前的多领域对话数据集，增添了对话管理的难度。CrossWOZ在网络爬取北京市的酒店/旅游景点/饭店以及地铁和出租车信息，并模拟用户向系统预订酒店景点等生成对话数据并进行标注。这里我们抽取其中双方对话意图作为意图分析分类数据集，本数据集为158分类的多标签意图分类数据集，数据集被划分为训练集、验证集、测试集。
（2）CrossWOZ数据集加载
将CrossWOZ数据集以及标签读取到内存，代码实现如下：
from paddlenlp.datasets import load_dataset
# 加载数据集标签
label_list = {}
label_map = {}
with open('dataset/label.txt', 'r', encoding='utf-8') as f:
for i, line in enumerate(f):
l = line.strip()
label_list[l] = i
label_map[i] = l
f.close()
# 定义数据加载函数
def load_data(path, label_list=None, is_test=False):
with open(path, 'r', encoding='utf-8') as f:
for line in f:
if is_test:
items = line.strip().split('\t')
sentence = ''.join(items)
yield {'sentence': sentence}
else:
items = line.strip().split('\t')
if len(items) == 0:
continue
elif len(items) == 1:
sentence = items[0]
labels = []
else:
sentence = ''.join(items[:-1])
label = items[-1]
labels = [label_list[l] for l in label.split(',')]
yield {'sentence': sentence, 'label': labels}
f.close()
train_dataset = load_dataset(load_data, path='dataset/train.txt', label_list=label_list, lazy=False)
dev_dataset = load_dataset(load_data, path='dataset/dev.txt', label_list=label_list, lazy=False)
test_dataset = load_dataset(load_data, path='dataset/test.txt', label_list=label_list, lazy=False)
print('训练集样本数量：', len(train_dataset))
print('验证集样本数量：', len(dev_dataset))
print('测试集样本数量：', len(test_dataset))
print('样本示例：', train_dataset[0])
训练集样本数量： 44409
验证集样本数量： 4935
测试集样本数量： 4909
样本示例： {'sentence': '你好，您可以选择故宫，八达岭长城，颐和园或者红砖美术馆。', 'label': [0]}
(3)数据预处理
数据预处理使用分词器（tokenizer）对文本进行切词并将每个词映射为分词器词典中的索引ID，并将数据集标签进行独热编码即One-Hot编码处理。索引ID方便模型后续根据这个ID找到该词对应的向量表示,ERNIE 3.0模型默认按中文字符对模型进行切词处理。
数据预处理代码实现如下：
import functools
def preprocess_function(examples, tokenizer, max_seq_len, num_classes, is_test=False):
encoded_inputs = tokenizer(text=examples["sentence"], max_seq_len=max_seq_len)
# One-Hot label
if not is_test:
encoded_inputs["labels"] = [float(1) if i in examples["label"] else float(0) for i in range(num_classes)]
return encoded_inputs
convert_example_to_feature = functools.partial(preprocess_function,
tokenizer=tokenizer,
max_seq_len=128,
num_classes=len(label_list))
# 训练集和开发集数据预处理，文字转为对应的ID，标签转为One Hot形式
train_dataset = train_dataset.map(convert_example_to_feature)
dev_dataset = dev_dataset.map(convert_example_to_feature)
test_dataset = test_dataset.map(convert_example_to_feature)
# 打印预处理后的数据样例
print('样本示例：', train_dataset[0])
样本示例： {'input_ids': [1, 226, 170, 4, 892, 48, 22, 352, 790, 470, 915, 4, 643, 302, 1560, 84, 257, 4, 3540, 14, 509, 172, 123, 536, 1520, 188, 133, 774, 12043, 2], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'labels': [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]}
从输出结果可以看到，文本串在分词后，已经将每个单词转换成了字典ID并且标签已经转为One Hot形式编码。
（4）构造DataLoader
在训练模型时，通常将数据分批传入模型进行训练，每批数据作为一个minibatch传入模型进行计算处理，每个minibatch数据包含两部分：文本数据和对应的意图分类标签，可以使用飞桨框架中的paddle.io.DataLoader实现批量迭代数据的功能。
from paddle.io import DataLoader, BatchSampler
from paddlenlp.data import DataCollatorWithPadding
# 文本填充
collate_fn = DataCollatorWithPadding(tokenizer)
# 构造batch sampler
train_batch_sampler = BatchSampler(train_dataset, batch_size=32, shuffle=True)
dev_batch_sampler = BatchSampler(dev_dataset, batch_size=32, shuffle=False)
test_batch_sampler = BatchSampler(test_dataset, batch_size=32, shuffle=False)
train_loader = DataLoader(dataset=train_dataset, batch_sampler=train_batch_sampler, collate_fn=collate_fn)
dev_loader = DataLoader(dataset=dev_dataset, batch_sampler=dev_batch_sampler, collate_fn=collate_fn)
test_loader = DataLoader(dataset=test_dataset, batch_sampler=test_batch_sampler, collate_fn=collate_fn)
5.训练配置
定义模型训练时用到的计算资源、模型、优化器、损失函数和优化指标等。
（1）训练参数。
（2）优化器：Adam优化器。
（3）损失函数：二元交叉熵（Binary Cross-Entropy）。
（4）评估指标：Macro F1和Micro F1。
代码实现如下：
import os
import paddle
from metric import MetricReport
# 定义训练参数
num_epochs = 30
log_steps = 50
eval_steps = 500
learning_rate = 0.00003
save_dir = "./checkpoints"
if not os.path.exists(save_dir):
os.makedirs(save_dir)
# 指定优化策略
optimizer = paddle.optimizer.Adam(learning_rate=learning_rate, parameters= model.parameters())
# 指定损失函数
loss_fn = paddle.nn.BCEWithLogitsLoss()
# 指定评估指标
metric = MetricReport()
6. 模型训练
在训练过程中，每隔log_steps打印一条训练日志，每隔eval_steps在验证集上进行一次模型评估，并且保存在训练过程中评估效果Micro F1最好模型。
在验证集上进行评估的代码实现如下：
import numpy as np
import paddle.nn.functional as F
@paddle.no_grad()
def evaluate(model, data_loader, metric):
# 将模型设置为评估模式
model.eval()
# 重置评价
metric.reset()
# 遍历验证集每个批次
for batch in data_loader:
labels = batch.pop("labels")
# 计算模型输出
logits = model(**batch)
probs = F.sigmoid(logits)
# 累积评价
metric.update(probs, labels)
micro_f1_score, macro_f1_score = metric.accumulate()
model.train()
metric.reset()
return micro_f1_score, macro_f1_score
训练模型的代码如下：
def train(model):
# 开启模型训练模式
model.train()
global_step = 0
best_score = 0.
# 记录训练过程中的损失函数值和验证集上的Micro F1
train_loss_record = []
train_score_record = []
num_training_steps = len(train_loader) * num_epochs
# 进行num_epochs轮训练
for epoch in range(num_epochs):
for step, batch in enumerate(train_loader):
labels = batch.pop("labels")
logits = model(**batch)
loss = loss_fn(logits, labels)
train_loss_record.append((global_step, loss.item()))
# 梯度反向传播
loss.backward()
optimizer.step()
optimizer.clear_grad()
if global_step > 0 and global_step % log_steps == 0:
print(f"- epoch: {epoch} - global_step: {global_step}/{num_training_steps} -loss: {loss.numpy().item():.6f}")
if global_step > 0 and global_step % eval_steps == 0:
micro_f1_score, macro_f1_score = evaluate(model, dev_loader, metric)
train_score_record.append((global_step, macro_f1_score))
if macro_f1_score > best_score:
print(f"best F1 performance has been updated: {best_score :.5f} --> {macro_f1_score :.5f}")
best_score = macro_f1_score
paddle.save(model.state_dict(), f"{save_dir}/best.pdparams")
print(f"evaluation result: Micro F1 score: {micro_f1_score:.5f}, Macro F1 score: {macro_f1_score:.5f}")
global_step += 1
paddle.save(model.state_dict(), f"{save_dir}/final.pdparams")
return train_loss_record, train_score_record
train_loss_record, train_score_record = train(model)
- epoch: 0 - global_step: 50/41640 -loss: 0.234528
- epoch: 0 - global_step: 100/41640 -loss: 0.108822
- epoch: 0 - global_step: 150/41640 -loss: 0.074900
- epoch: 0 - global_step: 200/41640 -loss: 0.058289
- epoch: 0 - global_step: 250/41640 -loss: 0.051151
- epoch: 0 - global_step: 300/41640 -loss: 0.054073
- epoch: 0 - global_step: 350/41640 -loss: 0.048517
- epoch: 0 - global_step: 400/41640 -loss: 0.045494
- epoch: 0 - global_step: 450/41640 -loss: 0.042297
- epoch: 0 - global_step: 500/41640 -loss: 0.042204
evaluation result: Micro F1 score: 0.00000, Macro F1 score: 0.00000
- epoch: 0 - global_step: 550/41640 -loss: 0.042710
- epoch: 0 - global_step: 600/41640 -loss: 0.038428
- epoch: 0 - global_step: 650/41640 -loss: 0.039055
- epoch: 0 - global_step: 700/41640 -loss: 0.037134
- epoch: 0 - global_step: 750/41640 -loss: 0.035417
- epoch: 0 - global_step: 800/41640 -loss: 0.038007
- epoch: 0 - global_step: 850/41640 -loss: 0.035113
- epoch: 0 - global_step: 900/41640 -loss: 0.028602
- epoch: 0 - global_step: 950/41640 -loss: 0.034285
- epoch: 0 - global_step: 1000/41640 -loss: 0.029837
best F1 performance has been updated: 0.00000 --> 0.03019
evaluation result: Micro F1 score: 0.47923, Macro F1 score: 0.03019
- epoch: 0 - global_step: 1050/41640 -loss: 0.031279
- epoch: 0 - global_step: 1100/41640 -loss: 0.024610
- epoch: 0 - global_step: 1150/41640 -loss: 0.027718
- epoch: 0 - global_step: 1200/41640 -loss: 0.025384
- epoch: 0 - global_step: 1250/41640 -loss: 0.028343
- epoch: 0 - global_step: 1300/41640 -loss: 0.024350
- epoch: 0 - global_step: 1350/41640 -loss: 0.020018
- epoch: 1 - global_step: 1400/41640 -loss: 0.020161
- epoch: 1 - global_step: 1450/41640 -loss: 0.017844
- epoch: 1 - global_step: 1500/41640 -loss: 0.019162
best F1 performance has been updated: 0.03019 --> 0.09975
evaluation result: Micro F1 score: 0.68366, Macro F1 score: 0.09975
- epoch: 1 - global_step: 1550/41640 -loss: 0.019093
- epoch: 1 - global_step: 1600/41640 -loss: 0.014747
- epoch: 1 - global_step: 1650/41640 -loss: 0.012942
- epoch: 1 - global_step: 1700/41640 -loss: 0.019696
- epoch: 1 - global_step: 1750/41640 -loss: 0.017540
- epoch: 1 - global_step: 1800/41640 -loss: 0.012733
- epoch: 1 - global_step: 1850/41640 -loss: 0.014289
- epoch: 1 - global_step: 1900/41640 -loss: 0.013409
- epoch: 1 - global_step: 1950/41640 -loss: 0.014473
- epoch: 1 - global_step: 2000/41640 -loss: 0.014607
best F1 performance has been updated: 0.09975 --> 0.19496
evaluation result: Micro F1 score: 0.83491, Macro F1 score: 0.19496
- epoch: 1 - global_step: 2050/41640 -loss: 0.012935
- epoch: 1 - global_step: 2100/41640 -loss: 0.013258
- epoch: 1 - global_step: 2150/41640 -loss: 0.011757
- epoch: 1 - global_step: 2200/41640 -loss: 0.009369
- epoch: 1 - global_step: 2250/41640 -loss: 0.009815
- epoch: 1 - global_step: 2300/41640 -loss: 0.010194
- epoch: 1 - global_step: 2350/41640 -loss: 0.011338
- epoch: 1 - global_step: 2400/41640 -loss: 0.007474
- epoch: 1 - global_step: 2450/41640 -loss: 0.009181
- epoch: 1 - global_step: 2500/41640 -loss: 0.010378
best F1 performance has been updated: 0.19496 --> 0.22206
evaluation result: Micro F1 score: 0.87484, Macro F1 score: 0.22206
- epoch: 1 - global_step: 2550/41640 -loss: 0.012969
- epoch: 1 - global_step: 2600/41640 -loss: 0.011587
- epoch: 1 - global_step: 2650/41640 -loss: 0.012332
- epoch: 1 - global_step: 2700/41640 -loss: 0.005341
- epoch: 1 - global_step: 2750/41640 -loss: 0.009883
- epoch: 2 - global_step: 2800/41640 -loss: 0.006675
- epoch: 2 - global_step: 2850/41640 -loss: 0.008585
- epoch: 2 - global_step: 2900/41640 -loss: 0.005240
- epoch: 2 - global_step: 2950/41640 -loss: 0.011724
- epoch: 2 - global_step: 3000/41640 -loss: 0.010401
best F1 performance has been updated: 0.22206 --> 0.24768
evaluation result: Micro F1 score: 0.89617, Macro F1 score: 0.24768
- epoch: 2 - global_step: 3050/41640 -loss: 0.010133
- epoch: 2 - global_step: 3100/41640 -loss: 0.003216
- epoch: 2 - global_step: 3150/41640 -loss: 0.005800
- epoch: 2 - global_step: 3200/41640 -loss: 0.005969
- epoch: 2 - global_step: 3250/41640 -loss: 0.005024
- epoch: 2 - global_step: 3300/41640 -loss: 0.010216
- epoch: 2 - global_step: 3350/41640 -loss: 0.007276
- epoch: 2 - global_step: 3400/41640 -loss: 0.009895
- epoch: 2 - global_step: 3450/41640 -loss: 0.007120
- epoch: 2 - global_step: 3500/41640 -loss: 0.007014
best F1 performance has been updated: 0.24768 --> 0.25819
evaluation result: Micro F1 score: 0.90136, Macro F1 score: 0.25819
- epoch: 2 - global_step: 3550/41640 -loss: 0.007284
- epoch: 2 - global_step: 3600/41640 -loss: 0.003644
- epoch: 2 - global_step: 3650/41640 -loss: 0.008800
- epoch: 2 - global_step: 3700/41640 -loss: 0.010350
- epoch: 2 - global_step: 3750/41640 -loss: 0.003111
- epoch: 2 - global_step: 3800/41640 -loss: 0.006959
- epoch: 2 - global_step: 3850/41640 -loss: 0.006032
- epoch: 2 - global_step: 3900/41640 -loss: 0.009836
- epoch: 2 - global_step: 3950/41640 -loss: 0.006473
- epoch: 2 - global_step: 4000/41640 -loss: 0.003897
best F1 performance has been updated: 0.25819 --> 0.28907
evaluation result: Micro F1 score: 0.91206, Macro F1 score: 0.28907
- epoch: 2 - global_step: 4050/41640 -loss: 0.006972
- epoch: 2 - global_step: 4100/41640 -loss: 0.011289
- epoch: 2 - global_step: 4150/41640 -loss: 0.009039
- epoch: 3 - global_step: 4200/41640 -loss: 0.004191
- epoch: 3 - global_step: 4250/41640 -loss: 0.006555
- epoch: 3 - global_step: 4300/41640 -loss: 0.005895
- epoch: 3 - global_step: 4350/41640 -loss: 0.005757
- epoch: 3 - global_step: 4400/41640 -loss: 0.005252
- epoch: 3 - global_step: 4450/41640 -loss: 0.002632
- epoch: 3 - global_step: 4500/41640 -loss: 0.007078
best F1 performance has been updated: 0.28907 --> 0.29607
evaluation result: Micro F1 score: 0.91456, Macro F1 score: 0.29607
- epoch: 3 - global_step: 4550/41640 -loss: 0.002729
- epoch: 3 - global_step: 4600/41640 -loss: 0.005051
- epoch: 3 - global_step: 4650/41640 -loss: 0.006038
- epoch: 3 - global_step: 4700/41640 -loss: 0.005034
- epoch: 3 - global_step: 4750/41640 -loss: 0.006570
- epoch: 3 - global_step: 4800/41640 -loss: 0.005794
- epoch: 3 - global_step: 4850/41640 -loss: 0.005689
- epoch: 3 - global_step: 4900/41640 -loss: 0.005141
- epoch: 3 - global_step: 4950/41640 -loss: 0.004101
- epoch: 3 - global_step: 5000/41640 -loss: 0.004738
best F1 performance has been updated: 0.29607 --> 0.29816
evaluation result: Micro F1 score: 0.91211, Macro F1 score: 0.29816
- epoch: 3 - global_step: 5050/41640 -loss: 0.004142
- epoch: 3 - global_step: 5100/41640 -loss: 0.007093
- epoch: 3 - global_step: 5150/41640 -loss: 0.007984
- epoch: 3 - global_step: 5200/41640 -loss: 0.003584
- epoch: 3 - global_step: 5250/41640 -loss: 0.007828
- epoch: 3 - global_step: 5300/41640 -loss: 0.003857
- epoch: 3 - global_step: 5350/41640 -loss: 0.005654
- epoch: 3 - global_step: 5400/41640 -loss: 0.003506
- epoch: 3 - global_step: 5450/41640 -loss: 0.007375
- epoch: 3 - global_step: 5500/41640 -loss: 0.004574
best F1 performance has been updated: 0.29816 --> 0.31497
evaluation result: Micro F1 score: 0.92147, Macro F1 score: 0.31497
- epoch: 3 - global_step: 5550/41640 -loss: 0.004191
- epoch: 4 - global_step: 5600/41640 -loss: 0.008325
- epoch: 4 - global_step: 5650/41640 -loss: 0.003151
- epoch: 4 - global_step: 5700/41640 -loss: 0.003114
- epoch: 4 - global_step: 5750/41640 -loss: 0.004714
- epoch: 4 - global_step: 5800/41640 -loss: 0.002818
- epoch: 4 - global_step: 5850/41640 -loss: 0.005083
- epoch: 4 - global_step: 5900/41640 -loss: 0.005644
- epoch: 4 - global_step: 5950/41640 -loss: 0.003744
- epoch: 4 - global_step: 6000/41640 -loss: 0.003304
best F1 performance has been updated: 0.31497 --> 0.33242
evaluation result: Micro F1 score: 0.92401, Macro F1 score: 0.33242
- epoch: 4 - global_step: 6050/41640 -loss: 0.002430
- epoch: 4 - global_step: 6100/41640 -loss: 0.002391
- epoch: 4 - global_step: 6150/41640 -loss: 0.004396
- epoch: 4 - global_step: 6200/41640 -loss: 0.004169
- epoch: 4 - global_step: 6250/41640 -loss: 0.006451
- epoch: 4 - global_step: 6300/41640 -loss: 0.006668
- epoch: 4 - global_step: 6350/41640 -loss: 0.003284
- epoch: 4 - global_step: 6400/41640 -loss: 0.001877
- epoch: 4 - global_step: 6450/41640 -loss: 0.007117
- epoch: 4 - global_step: 6500/41640 -loss: 0.005236
best F1 performance has been updated: 0.33242 --> 0.33291
evaluation result: Micro F1 score: 0.92389, Macro F1 score: 0.33291
- epoch: 4 - global_step: 6550/41640 -loss: 0.002663
- epoch: 4 - global_step: 6600/41640 -loss: 0.004751
- epoch: 4 - global_step: 6650/41640 -loss: 0.003015
- epoch: 4 - global_step: 6700/41640 -loss: 0.004396
- epoch: 4 - global_step: 6750/41640 -loss: 0.002194
- epoch: 4 - global_step: 6800/41640 -loss: 0.004072
- epoch: 4 - global_step: 6850/41640 -loss: 0.004502
- epoch: 4 - global_step: 6900/41640 -loss: 0.005123
- epoch: 5 - global_step: 6950/41640 -loss: 0.005744
- epoch: 5 - global_step: 7000/41640 -loss: 0.001654
best F1 performance has been updated: 0.33291 --> 0.33800
evaluation result: Micro F1 score: 0.92331, Macro F1 score: 0.33800
- epoch: 5 - global_step: 7050/41640 -loss: 0.005241
- epoch: 5 - global_step: 7100/41640 -loss: 0.007419
- epoch: 5 - global_step: 7150/41640 -loss: 0.001067
- epoch: 5 - global_step: 7200/41640 -loss: 0.001657
- epoch: 5 - global_step: 7250/41640 -loss: 0.001502
- epoch: 5 - global_step: 7300/41640 -loss: 0.002529
- epoch: 5 - global_step: 7350/41640 -loss: 0.004248
- epoch: 5 - global_step: 7400/41640 -loss: 0.001019
- epoch: 5 - global_step: 7450/41640 -loss: 0.002655
- epoch: 5 - global_step: 7500/41640 -loss: 0.003435
best F1 performance has been updated: 0.33800 --> 0.34757
evaluation result: Micro F1 score: 0.92705, Macro F1 score: 0.34757
- epoch: 5 - global_step: 7550/41640 -loss: 0.002648
- epoch: 5 - global_step: 7600/41640 -loss: 0.003658
- epoch: 5 - global_step: 7650/41640 -loss: 0.007714
- epoch: 5 - global_step: 7700/41640 -loss: 0.002942
- epoch: 5 - global_step: 7750/41640 -loss: 0.004334
- epoch: 5 - global_step: 7800/41640 -loss: 0.002246
- epoch: 5 - global_step: 7850/41640 -loss: 0.003070
- epoch: 5 - global_step: 7900/41640 -loss: 0.002987
- epoch: 5 - global_step: 7950/41640 -loss: 0.006196
- epoch: 5 - global_step: 8000/41640 -loss: 0.004129
best F1 performance has been updated: 0.34757 --> 0.35756
evaluation result: Micro F1 score: 0.92589, Macro F1 score: 0.35756
- epoch: 5 - global_step: 8050/41640 -loss: 0.004331
- epoch: 5 - global_step: 8100/41640 -loss: 0.002497
- epoch: 5 - global_step: 8150/41640 -loss: 0.003752
- epoch: 5 - global_step: 8200/41640 -loss: 0.006582
- epoch: 5 - global_step: 8250/41640 -loss: 0.003617
- epoch: 5 - global_step: 8300/41640 -loss: 0.001648
- epoch: 6 - global_step: 8350/41640 -loss: 0.002741
- epoch: 6 - global_step: 8400/41640 -loss: 0.002592
- epoch: 6 - global_step: 8450/41640 -loss: 0.001317
- epoch: 6 - global_step: 8500/41640 -loss: 0.003566
evaluation result: Micro F1 score: 0.92830, Macro F1 score: 0.35401
- epoch: 6 - global_step: 8550/41640 -loss: 0.001996
- epoch: 6 - global_step: 8600/41640 -loss: 0.004015
- epoch: 6 - global_step: 8650/41640 -loss: 0.001388
- epoch: 6 - global_step: 8700/41640 -loss: 0.002029
- epoch: 6 - global_step: 8750/41640 -loss: 0.004452
- epoch: 6 - global_step: 8800/41640 -loss: 0.003538
- epoch: 6 - global_step: 8850/41640 -loss: 0.003933
- epoch: 6 - global_step: 8900/41640 -loss: 0.002802
- epoch: 6 - global_step: 8950/41640 -loss: 0.004460
- epoch: 6 - global_step: 9000/41640 -loss: 0.002321
best F1 performance has been updated: 0.35756 --> 0.37876
evaluation result: Micro F1 score: 0.92950, Macro F1 score: 0.37876
- epoch: 6 - global_step: 9050/41640 -loss: 0.002876
- epoch: 6 - global_step: 9100/41640 -loss: 0.002835
- epoch: 6 - global_step: 9150/41640 -loss: 0.000683
- epoch: 6 - global_step: 9200/41640 -loss: 0.000843
- epoch: 6 - global_step: 9250/41640 -loss: 0.003733
- epoch: 6 - global_step: 9300/41640 -loss: 0.005470
- epoch: 6 - global_step: 9350/41640 -loss: 0.003148
- epoch: 6 - global_step: 9400/41640 -loss: 0.001495
- epoch: 6 - global_step: 9450/41640 -loss: 0.001947
- epoch: 6 - global_step: 9500/41640 -loss: 0.002560
evaluation result: Micro F1 score: 0.93017, Macro F1 score: 0.37842
- epoch: 6 - global_step: 9550/41640 -loss: 0.001466
- epoch: 6 - global_step: 9600/41640 -loss: 0.003991
- epoch: 6 - global_step: 9650/41640 -loss: 0.002581
- epoch: 6 - global_step: 9700/41640 -loss: 0.002148
- epoch: 7 - global_step: 9750/41640 -loss: 0.002497
- epoch: 7 - global_step: 9800/41640 -loss: 0.005755
- epoch: 7 - global_step: 9850/41640 -loss: 0.000927
- epoch: 7 - global_step: 9900/41640 -loss: 0.005453
- epoch: 7 - global_step: 9950/41640 -loss: 0.002838
- epoch: 7 - global_step: 10000/41640 -loss: 0.001353
best F1 performance has been updated: 0.37876 --> 0.39624
evaluation result: Micro F1 score: 0.93013, Macro F1 score: 0.39624
- epoch: 7 - global_step: 10050/41640 -loss: 0.002071
- epoch: 7 - global_step: 10100/41640 -loss: 0.003925
- epoch: 7 - global_step: 10150/41640 -loss: 0.001539
- epoch: 7 - global_step: 10200/41640 -loss: 0.002633
- epoch: 7 - global_step: 10250/41640 -loss: 0.002811
- epoch: 7 - global_step: 10300/41640 -loss: 0.002451
- epoch: 7 - global_step: 10350/41640 -loss: 0.004894
- epoch: 7 - global_step: 10400/41640 -loss: 0.000807
- epoch: 7 - global_step: 10450/41640 -loss: 0.001253
- epoch: 7 - global_step: 10500/41640 -loss: 0.001653
evaluation result: Micro F1 score: 0.93144, Macro F1 score: 0.39484
- epoch: 7 - global_step: 10550/41640 -loss: 0.003801
- epoch: 7 - global_step: 10600/41640 -loss: 0.002032
- epoch: 7 - global_step: 10650/41640 -loss: 0.002959
- epoch: 7 - global_step: 10700/41640 -loss: 0.001480
- epoch: 7 - global_step: 10750/41640 -loss: 0.002376
- epoch: 7 - global_step: 10800/41640 -loss: 0.004908
- epoch: 7 - global_step: 10850/41640 -loss: 0.003244
- epoch: 7 - global_step: 10900/41640 -loss: 0.003521
- epoch: 7 - global_step: 10950/41640 -loss: 0.000739
- epoch: 7 - global_step: 11000/41640 -loss: 0.003595
best F1 performance has been updated: 0.39624 --> 0.43173
evaluation result: Micro F1 score: 0.93305, Macro F1 score: 0.43173
- epoch: 7 - global_step: 11050/41640 -loss: 0.003875
- epoch: 7 - global_step: 11100/41640 -loss: 0.000711
- epoch: 8 - global_step: 11150/41640 -loss: 0.000824
- epoch: 8 - global_step: 11200/41640 -loss: 0.002089
- epoch: 8 - global_step: 11250/41640 -loss: 0.000834
- epoch: 8 - global_step: 11300/41640 -loss: 0.001334
- epoch: 8 - global_step: 11350/41640 -loss: 0.002950
- epoch: 8 - global_step: 11400/41640 -loss: 0.003803
- epoch: 8 - global_step: 11450/41640 -loss: 0.000318
- epoch: 8 - global_step: 11500/41640 -loss: 0.001495
best F1 performance has been updated: 0.43173 --> 0.44560
evaluation result: Micro F1 score: 0.93342, Macro F1 score: 0.44560
- epoch: 8 - global_step: 11550/41640 -loss: 0.001434
- epoch: 8 - global_step: 11600/41640 -loss: 0.001734
- epoch: 8 - global_step: 11650/41640 -loss: 0.000878
- epoch: 8 - global_step: 11700/41640 -loss: 0.003366
- epoch: 8 - global_step: 11750/41640 -loss: 0.000875
- epoch: 8 - global_step: 11800/41640 -loss: 0.001657
- epoch: 8 - global_step: 11850/41640 -loss: 0.003274
- epoch: 8 - global_step: 11900/41640 -loss: 0.000821
- epoch: 8 - global_step: 11950/41640 -loss: 0.002623
- epoch: 8 - global_step: 12000/41640 -loss: 0.000869
evaluation result: Micro F1 score: 0.93144, Macro F1 score: 0.44160
- epoch: 8 - global_step: 12050/41640 -loss: 0.002223
- epoch: 8 - global_step: 12100/41640 -loss: 0.002583
- epoch: 8 - global_step: 12150/41640 -loss: 0.001624
- epoch: 8 - global_step: 12200/41640 -loss: 0.001837
- epoch: 8 - global_step: 12250/41640 -loss: 0.004740
- epoch: 8 - global_step: 12300/41640 -loss: 0.001465
- epoch: 8 - global_step: 12350/41640 -loss: 0.003284
- epoch: 8 - global_step: 12400/41640 -loss: 0.001100
- epoch: 8 - global_step: 12450/41640 -loss: 0.002647
- epoch: 9 - global_step: 12500/41640 -loss: 0.003348
best F1 performance has been updated: 0.44560 --> 0.51015
evaluation result: Micro F1 score: 0.93221, Macro F1 score: 0.51015
- epoch: 9 - global_step: 12550/41640 -loss: 0.001193
- epoch: 9 - global_step: 12600/41640 -loss: 0.001739
- epoch: 9 - global_step: 12650/41640 -loss: 0.001326
- epoch: 9 - global_step: 12700/41640 -loss: 0.001472
- epoch: 9 - global_step: 12750/41640 -loss: 0.003599
- epoch: 9 - global_step: 12800/41640 -loss: 0.003130
- epoch: 9 - global_step: 12850/41640 -loss: 0.001368
- epoch: 9 - global_step: 12900/41640 -loss: 0.001877
- epoch: 9 - global_step: 12950/41640 -loss: 0.000868
- epoch: 9 - global_step: 13000/41640 -loss: 0.002654
evaluation result: Micro F1 score: 0.93415, Macro F1 score: 0.48296
- epoch: 9 - global_step: 13050/41640 -loss: 0.003214
- epoch: 9 - global_step: 13100/41640 -loss: 0.000414
- epoch: 9 - global_step: 13150/41640 -loss: 0.000389
- epoch: 9 - global_step: 13200/41640 -loss: 0.002357
- epoch: 9 - global_step: 13250/41640 -loss: 0.000353
- epoch: 9 - global_step: 13300/41640 -loss: 0.005102
- epoch: 9 - global_step: 13350/41640 -loss: 0.001201
- epoch: 9 - global_step: 13400/41640 -loss: 0.003059
- epoch: 9 - global_step: 13450/41640 -loss: 0.005206
- epoch: 9 - global_step: 13500/41640 -loss: 0.002283
best F1 performance has been updated: 0.51015 --> 0.53404
evaluation result: Micro F1 score: 0.93438, Macro F1 score: 0.53404
- epoch: 9 - global_step: 13550/41640 -loss: 0.000883
- epoch: 9 - global_step: 13600/41640 -loss: 0.003967
- epoch: 9 - global_step: 13650/41640 -loss: 0.001158
- epoch: 9 - global_step: 13700/41640 -loss: 0.001351
- epoch: 9 - global_step: 13750/41640 -loss: 0.000896
- epoch: 9 - global_step: 13800/41640 -loss: 0.004641
- epoch: 9 - global_step: 13850/41640 -loss: 0.004213
- epoch: 10 - global_step: 13900/41640 -loss: 0.001853
- epoch: 10 - global_step: 13950/41640 -loss: 0.003138
- epoch: 10 - global_step: 14000/41640 -loss: 0.000631
best F1 performance has been updated: 0.53404 --> 0.54464
evaluation result: Micro F1 score: 0.93715, Macro F1 score: 0.54464
- epoch: 10 - global_step: 14050/41640 -loss: 0.003507
- epoch: 10 - global_step: 14100/41640 -loss: 0.002514
- epoch: 10 - global_step: 14150/41640 -loss: 0.000901
- epoch: 10 - global_step: 14200/41640 -loss: 0.001694
- epoch: 10 - global_step: 14250/41640 -loss: 0.002222
- epoch: 10 - global_step: 14300/41640 -loss: 0.003585
- epoch: 10 - global_step: 14350/41640 -loss: 0.001025
- epoch: 10 - global_step: 14400/41640 -loss: 0.001434
- epoch: 10 - global_step: 14450/41640 -loss: 0.000822
- epoch: 10 - global_step: 14500/41640 -loss: 0.001143
evaluation result: Micro F1 score: 0.93476, Macro F1 score: 0.53605
- epoch: 10 - global_step: 14550/41640 -loss: 0.001757
- epoch: 10 - global_step: 14600/41640 -loss: 0.002667
- epoch: 10 - global_step: 14650/41640 -loss: 0.001649
- epoch: 10 - global_step: 14700/41640 -loss: 0.001299
- epoch: 10 - global_step: 14750/41640 -loss: 0.004475
- epoch: 10 - global_step: 14800/41640 -loss: 0.001884
- epoch: 10 - global_step: 14850/41640 -loss: 0.000935
- epoch: 10 - global_step: 14900/41640 -loss: 0.000548
- epoch: 10 - global_step: 14950/41640 -loss: 0.000990
- epoch: 10 - global_step: 15000/41640 -loss: 0.004166
best F1 performance has been updated: 0.54464 --> 0.54986
evaluation result: Micro F1 score: 0.93661, Macro F1 score: 0.54986
- epoch: 10 - global_step: 15050/41640 -loss: 0.001219
- epoch: 10 - global_step: 15100/41640 -loss: 0.002339
- epoch: 10 - global_step: 15150/41640 -loss: 0.003213
- epoch: 10 - global_step: 15200/41640 -loss: 0.001149
- epoch: 10 - global_step: 15250/41640 -loss: 0.001783
- epoch: 11 - global_step: 15300/41640 -loss: 0.002059
- epoch: 11 - global_step: 15350/41640 -loss: 0.001683
- epoch: 11 - global_step: 15400/41640 -loss: 0.003432
- epoch: 11 - global_step: 15450/41640 -loss: 0.001163
- epoch: 11 - global_step: 15500/41640 -loss: 0.000398
best F1 performance has been updated: 0.54986 --> 0.56374
evaluation result: Micro F1 score: 0.93654, Macro F1 score: 0.56374
- epoch: 11 - global_step: 15550/41640 -loss: 0.001650
- epoch: 11 - global_step: 15600/41640 -loss: 0.002306
- epoch: 11 - global_step: 15650/41640 -loss: 0.001242
- epoch: 11 - global_step: 15700/41640 -loss: 0.001438
- epoch: 11 - global_step: 15750/41640 -loss: 0.000227
- epoch: 11 - global_step: 15800/41640 -loss: 0.000898
- epoch: 11 - global_step: 15850/41640 -loss: 0.002197
- epoch: 11 - global_step: 15900/41640 -loss: 0.002972
- epoch: 11 - global_step: 15950/41640 -loss: 0.000830
- epoch: 11 - global_step: 16000/41640 -loss: 0.000442
evaluation result: Micro F1 score: 0.93504, Macro F1 score: 0.56343
- epoch: 11 - global_step: 16050/41640 -loss: 0.000288
- epoch: 11 - global_step: 16100/41640 -loss: 0.001795
- epoch: 11 - global_step: 16150/41640 -loss: 0.003863
- epoch: 11 - global_step: 16200/41640 -loss: 0.000663
- epoch: 11 - global_step: 16250/41640 -loss: 0.001006
- epoch: 11 - global_step: 16300/41640 -loss: 0.001142
- epoch: 11 - global_step: 16350/41640 -loss: 0.001224
- epoch: 11 - global_step: 16400/41640 -loss: 0.001641
- epoch: 11 - global_step: 16450/41640 -loss: 0.004579
- epoch: 11 - global_step: 16500/41640 -loss: 0.001029
evaluation result: Micro F1 score: 0.93499, Macro F1 score: 0.53820
- epoch: 11 - global_step: 16550/41640 -loss: 0.002012
- epoch: 11 - global_step: 16600/41640 -loss: 0.001568
- epoch: 11 - global_step: 16650/41640 -loss: 0.002507
- epoch: 12 - global_step: 16700/41640 -loss: 0.000933
- epoch: 12 - global_step: 16750/41640 -loss: 0.003079
- epoch: 12 - global_step: 16800/41640 -loss: 0.000570
- epoch: 12 - global_step: 16850/41640 -loss: 0.001927
- epoch: 12 - global_step: 16900/41640 -loss: 0.000659
- epoch: 12 - global_step: 16950/41640 -loss: 0.002229
- epoch: 12 - global_step: 17000/41640 -loss: 0.000725
best F1 performance has been updated: 0.56374 --> 0.57136
evaluation result: Micro F1 score: 0.93744, Macro F1 score: 0.57136
- epoch: 12 - global_step: 17050/41640 -loss: 0.000384
- epoch: 12 - global_step: 17100/41640 -loss: 0.002075
- epoch: 12 - global_step: 17150/41640 -loss: 0.001048
- epoch: 12 - global_step: 17200/41640 -loss: 0.001718
- epoch: 12 - global_step: 17250/41640 -loss: 0.000285
- epoch: 12 - global_step: 17300/41640 -loss: 0.001711
- epoch: 12 - global_step: 17350/41640 -loss: 0.002268
- epoch: 12 - global_step: 17400/41640 -loss: 0.002390
- epoch: 12 - global_step: 17450/41640 -loss: 0.003974
- epoch: 12 - global_step: 17500/41640 -loss: 0.001192
best F1 performance has been updated: 0.57136 --> 0.59027
evaluation result: Micro F1 score: 0.93461, Macro F1 score: 0.59027
- epoch: 12 - global_step: 17550/41640 -loss: 0.002425
- epoch: 12 - global_step: 17600/41640 -loss: 0.002325
- epoch: 12 - global_step: 17650/41640 -loss: 0.001513
- epoch: 12 - global_step: 17700/41640 -loss: 0.002475
- epoch: 12 - global_step: 17750/41640 -loss: 0.000882
- epoch: 12 - global_step: 17800/41640 -loss: 0.001909
- epoch: 12 - global_step: 17850/41640 -loss: 0.001645
- epoch: 12 - global_step: 17900/41640 -loss: 0.000949
- epoch: 12 - global_step: 17950/41640 -loss: 0.001218
- epoch: 12 - global_step: 18000/41640 -loss: 0.000796
best F1 performance has been updated: 0.59027 --> 0.61284
evaluation result: Micro F1 score: 0.93764, Macro F1 score: 0.61284
- epoch: 13 - global_step: 18050/41640 -loss: 0.001676
- epoch: 13 - global_step: 18100/41640 -loss: 0.000676
- epoch: 13 - global_step: 18150/41640 -loss: 0.002620
- epoch: 13 - global_step: 18200/41640 -loss: 0.001917
- epoch: 13 - global_step: 18250/41640 -loss: 0.002936
- epoch: 13 - global_step: 18300/41640 -loss: 0.002927
- epoch: 13 - global_step: 18350/41640 -loss: 0.001687
- epoch: 13 - global_step: 18400/41640 -loss: 0.001485
- epoch: 13 - global_step: 18450/41640 -loss: 0.000634
- epoch: 13 - global_step: 18500/41640 -loss: 0.001227
evaluation result: Micro F1 score: 0.93766, Macro F1 score: 0.58453
- epoch: 13 - global_step: 18550/41640 -loss: 0.000857
- epoch: 13 - global_step: 18600/41640 -loss: 0.000784
- epoch: 13 - global_step: 18650/41640 -loss: 0.000164
- epoch: 13 - global_step: 18700/41640 -loss: 0.001125
- epoch: 13 - global_step: 18750/41640 -loss: 0.001099
- epoch: 13 - global_step: 18800/41640 -loss: 0.001576
- epoch: 13 - global_step: 18850/41640 -loss: 0.001748
- epoch: 13 - global_step: 18900/41640 -loss: 0.001790
- epoch: 13 - global_step: 18950/41640 -loss: 0.001131
- epoch: 13 - global_step: 19000/41640 -loss: 0.000993
evaluation result: Micro F1 score: 0.93802, Macro F1 score: 0.60875
- epoch: 13 - global_step: 19050/41640 -loss: 0.000514
- epoch: 13 - global_step: 19100/41640 -loss: 0.002475
- epoch: 13 - global_step: 19150/41640 -loss: 0.001912
- epoch: 13 - global_step: 19200/41640 -loss: 0.002594
- epoch: 13 - global_step: 19250/41640 -loss: 0.001062
- epoch: 13 - global_step: 19300/41640 -loss: 0.001124
- epoch: 13 - global_step: 19350/41640 -loss: 0.002056
- epoch: 13 - global_step: 19400/41640 -loss: 0.001288
- epoch: 14 - global_step: 19450/41640 -loss: 0.000398
- epoch: 14 - global_step: 19500/41640 -loss: 0.000170
best F1 performance has been updated: 0.61284 --> 0.62442
evaluation result: Micro F1 score: 0.93910, Macro F1 score: 0.62442
- epoch: 14 - global_step: 19550/41640 -loss: 0.001000
- epoch: 14 - global_step: 19600/41640 -loss: 0.000866
- epoch: 14 - global_step: 19650/41640 -loss: 0.003032
- epoch: 14 - global_step: 19700/41640 -loss: 0.000443
- epoch: 14 - global_step: 19750/41640 -loss: 0.001522
- epoch: 14 - global_step: 19800/41640 -loss: 0.000231
- epoch: 14 - global_step: 19850/41640 -loss: 0.000437
- epoch: 14 - global_step: 19900/41640 -loss: 0.001140
- epoch: 14 - global_step: 19950/41640 -loss: 0.000779
- epoch: 14 - global_step: 20000/41640 -loss: 0.001300
best F1 performance has been updated: 0.62442 --> 0.63095
evaluation result: Micro F1 score: 0.93858, Macro F1 score: 0.63095
- epoch: 14 - global_step: 20050/41640 -loss: 0.001256
- epoch: 14 - global_step: 20100/41640 -loss: 0.002813
- epoch: 14 - global_step: 20150/41640 -loss: 0.000917
- epoch: 14 - global_step: 20200/41640 -loss: 0.004287
- epoch: 14 - global_step: 20250/41640 -loss: 0.001725
- epoch: 14 - global_step: 20300/41640 -loss: 0.000346
- epoch: 14 - global_step: 20350/41640 -loss: 0.000288
- epoch: 14 - global_step: 20400/41640 -loss: 0.001294
- epoch: 14 - global_step: 20450/41640 -loss: 0.005925
- epoch: 14 - global_step: 20500/41640 -loss: 0.000676
evaluation result: Micro F1 score: 0.94166, Macro F1 score: 0.61658
- epoch: 14 - global_step: 20550/41640 -loss: 0.004796
- epoch: 14 - global_step: 20600/41640 -loss: 0.000778
- epoch: 14 - global_step: 20650/41640 -loss: 0.002352
- epoch: 14 - global_step: 20700/41640 -loss: 0.001819
- epoch: 14 - global_step: 20750/41640 -loss: 0.000703
- epoch: 14 - global_step: 20800/41640 -loss: 0.001096
- epoch: 15 - global_step: 20850/41640 -loss: 0.000349
- epoch: 15 - global_step: 20900/41640 -loss: 0.001774
- epoch: 15 - global_step: 20950/41640 -loss: 0.001423
- epoch: 15 - global_step: 21000/41640 -loss: 0.003014
evaluation result: Micro F1 score: 0.93984, Macro F1 score: 0.61374
- epoch: 15 - global_step: 21050/41640 -loss: 0.000363
- epoch: 15 - global_step: 21100/41640 -loss: 0.000184
- epoch: 15 - global_step: 21150/41640 -loss: 0.000089
- epoch: 15 - global_step: 21200/41640 -loss: 0.001050
- epoch: 15 - global_step: 21250/41640 -loss: 0.001751
- epoch: 15 - global_step: 21300/41640 -loss: 0.001413
- epoch: 15 - global_step: 21350/41640 -loss: 0.000669
- epoch: 15 - global_step: 21400/41640 -loss: 0.000341
- epoch: 15 - global_step: 21450/41640 -loss: 0.001707
- epoch: 15 - global_step: 21500/41640 -loss: 0.001416
evaluation result: Micro F1 score: 0.93874, Macro F1 score: 0.61940
- epoch: 15 - global_step: 21550/41640 -loss: 0.002330
- epoch: 15 - global_step: 21600/41640 -loss: 0.000739
- epoch: 15 - global_step: 21650/41640 -loss: 0.002176
- epoch: 15 - global_step: 21700/41640 -loss: 0.003057
- epoch: 15 - global_step: 21750/41640 -loss: 0.000323
- epoch: 15 - global_step: 21800/41640 -loss: 0.001467
- epoch: 15 - global_step: 21850/41640 -loss: 0.003900
- epoch: 15 - global_step: 21900/41640 -loss: 0.000437
- epoch: 15 - global_step: 21950/41640 -loss: 0.001646
- epoch: 15 - global_step: 22000/41640 -loss: 0.001035
evaluation result: Micro F1 score: 0.94056, Macro F1 score: 0.62960
- epoch: 15 - global_step: 22050/41640 -loss: 0.002890
- epoch: 15 - global_step: 22100/41640 -loss: 0.002070
- epoch: 15 - global_step: 22150/41640 -loss: 0.002343
- epoch: 15 - global_step: 22200/41640 -loss: 0.002357
- epoch: 16 - global_step: 22250/41640 -loss: 0.000886
- epoch: 16 - global_step: 22300/41640 -loss: 0.001003
- epoch: 16 - global_step: 22350/41640 -loss: 0.000915
- epoch: 16 - global_step: 22400/41640 -loss: 0.002537
- epoch: 16 - global_step: 22450/41640 -loss: 0.000876
- epoch: 16 - global_step: 22500/41640 -loss: 0.000761
best F1 performance has been updated: 0.63095 --> 0.64101
evaluation result: Micro F1 score: 0.94047, Macro F1 score: 0.64101
- epoch: 16 - global_step: 22550/41640 -loss: 0.000690
- epoch: 16 - global_step: 22600/41640 -loss: 0.001432
- epoch: 16 - global_step: 22650/41640 -loss: 0.002738
- epoch: 16 - global_step: 22700/41640 -loss: 0.001512
- epoch: 16 - global_step: 22750/41640 -loss: 0.001159
- epoch: 16 - global_step: 22800/41640 -loss: 0.000671
- epoch: 16 - global_step: 22850/41640 -loss: 0.000772
- epoch: 16 - global_step: 22900/41640 -loss: 0.002284
- epoch: 16 - global_step: 22950/41640 -loss: 0.001978
- epoch: 16 - global_step: 23000/41640 -loss: 0.000814
best F1 performance has been updated: 0.64101 --> 0.64635
evaluation result: Micro F1 score: 0.93660, Macro F1 score: 0.64635
- epoch: 16 - global_step: 23050/41640 -loss: 0.002060
- epoch: 16 - global_step: 23100/41640 -loss: 0.001256
- epoch: 16 - global_step: 23150/41640 -loss: 0.001661
- epoch: 16 - global_step: 23200/41640 -loss: 0.000704
- epoch: 16 - global_step: 23250/41640 -loss: 0.003432
- epoch: 16 - global_step: 23300/41640 -loss: 0.001733
- epoch: 16 - global_step: 23350/41640 -loss: 0.000625
- epoch: 16 - global_step: 23400/41640 -loss: 0.000383
- epoch: 16 - global_step: 23450/41640 -loss: 0.000299
- epoch: 16 - global_step: 23500/41640 -loss: 0.004722
evaluation result: Micro F1 score: 0.93921, Macro F1 score: 0.64265
- epoch: 16 - global_step: 23550/41640 -loss: 0.001996
- epoch: 17 - global_step: 23600/41640 -loss: 0.001231
- epoch: 17 - global_step: 23650/41640 -loss: 0.000517
- epoch: 17 - global_step: 23700/41640 -loss: 0.000449
- epoch: 17 - global_step: 23750/41640 -loss: 0.002240
- epoch: 17 - global_step: 23800/41640 -loss: 0.000447
- epoch: 17 - global_step: 23850/41640 -loss: 0.003307
- epoch: 17 - global_step: 23900/41640 -loss: 0.002938
- epoch: 17 - global_step: 23950/41640 -loss: 0.000670
- epoch: 17 - global_step: 24000/41640 -loss: 0.000331
evaluation result: Micro F1 score: 0.93861, Macro F1 score: 0.61746
- epoch: 17 - global_step: 24050/41640 -loss: 0.000451
- epoch: 17 - global_step: 24100/41640 -loss: 0.000697
- epoch: 17 - global_step: 24150/41640 -loss: 0.002359
- epoch: 17 - global_step: 24200/41640 -loss: 0.001096
- epoch: 17 - global_step: 24250/41640 -loss: 0.003017
- epoch: 17 - global_step: 24300/41640 -loss: 0.000304
- epoch: 17 - global_step: 24350/41640 -loss: 0.000411
- epoch: 17 - global_step: 24400/41640 -loss: 0.001144
- epoch: 17 - global_step: 24450/41640 -loss: 0.001126
- epoch: 17 - global_step: 24500/41640 -loss: 0.001962
best F1 performance has been updated: 0.64635 --> 0.65768
evaluation result: Micro F1 score: 0.94102, Macro F1 score: 0.65768
- epoch: 17 - global_step: 24550/41640 -loss: 0.000926
- epoch: 17 - global_step: 24600/41640 -loss: 0.000571
- epoch: 17 - global_step: 24650/41640 -loss: 0.000813
- epoch: 17 - global_step: 24700/41640 -loss: 0.000121
- epoch: 17 - global_step: 24750/41640 -loss: 0.001049
- epoch: 17 - global_step: 24800/41640 -loss: 0.000340
- epoch: 17 - global_step: 24850/41640 -loss: 0.001074
- epoch: 17 - global_step: 24900/41640 -loss: 0.000092
- epoch: 17 - global_step: 24950/41640 -loss: 0.001688
- epoch: 18 - global_step: 25000/41640 -loss: 0.000343
evaluation result: Micro F1 score: 0.94093, Macro F1 score: 0.65768
- epoch: 18 - global_step: 25050/41640 -loss: 0.000616
- epoch: 18 - global_step: 25100/41640 -loss: 0.003129
- epoch: 18 - global_step: 25150/41640 -loss: 0.000577
- epoch: 18 - global_step: 25200/41640 -loss: 0.000109
- epoch: 18 - global_step: 25250/41640 -loss: 0.000994
- epoch: 18 - global_step: 25300/41640 -loss: 0.000972
- epoch: 18 - global_step: 25350/41640 -loss: 0.002141
- epoch: 18 - global_step: 25400/41640 -loss: 0.000632
- epoch: 18 - global_step: 25450/41640 -loss: 0.000638
- epoch: 18 - global_step: 25500/41640 -loss: 0.000701
best F1 performance has been updated: 0.65768 --> 0.68433
evaluation result: Micro F1 score: 0.94114, Macro F1 score: 0.68433
- epoch: 18 - global_step: 25550/41640 -loss: 0.001252
- epoch: 18 - global_step: 25600/41640 -loss: 0.003465
- epoch: 18 - global_step: 25650/41640 -loss: 0.001291
- epoch: 18 - global_step: 25700/41640 -loss: 0.000405
- epoch: 18 - global_step: 25750/41640 -loss: 0.000956
- epoch: 18 - global_step: 25800/41640 -loss: 0.001397
- epoch: 18 - global_step: 25850/41640 -loss: 0.000525
- epoch: 18 - global_step: 25900/41640 -loss: 0.000329
- epoch: 18 - global_step: 25950/41640 -loss: 0.004670
- epoch: 18 - global_step: 26000/41640 -loss: 0.002836
evaluation result: Micro F1 score: 0.93877, Macro F1 score: 0.66120
- epoch: 18 - global_step: 26050/41640 -loss: 0.002141
- epoch: 18 - global_step: 26100/41640 -loss: 0.000278
- epoch: 18 - global_step: 26150/41640 -loss: 0.000144
- epoch: 18 - global_step: 26200/41640 -loss: 0.003334
- epoch: 18 - global_step: 26250/41640 -loss: 0.001743
- epoch: 18 - global_step: 26300/41640 -loss: 0.000077
- epoch: 18 - global_step: 26350/41640 -loss: 0.002859
- epoch: 19 - global_step: 26400/41640 -loss: 0.000075
- epoch: 19 - global_step: 26450/41640 -loss: 0.000314
- epoch: 19 - global_step: 26500/41640 -loss: 0.000323
evaluation result: Micro F1 score: 0.93840, Macro F1 score: 0.67013
- epoch: 19 - global_step: 26550/41640 -loss: 0.001773
- epoch: 19 - global_step: 26600/41640 -loss: 0.001354
- epoch: 19 - global_step: 26650/41640 -loss: 0.001779
- epoch: 19 - global_step: 26700/41640 -loss: 0.000897
- epoch: 19 - global_step: 26750/41640 -loss: 0.000499
- epoch: 19 - global_step: 26800/41640 -loss: 0.000118
- epoch: 19 - global_step: 26850/41640 -loss: 0.000609
- epoch: 19 - global_step: 26900/41640 -loss: 0.000757
- epoch: 19 - global_step: 26950/41640 -loss: 0.000303
- epoch: 19 - global_step: 27000/41640 -loss: 0.002011
best F1 performance has been updated: 0.68433 --> 0.68471
evaluation result: Micro F1 score: 0.93893, Macro F1 score: 0.68471
- epoch: 19 - global_step: 27050/41640 -loss: 0.000542
- epoch: 19 - global_step: 27100/41640 -loss: 0.000344
- epoch: 19 - global_step: 27150/41640 -loss: 0.000468
- epoch: 19 - global_step: 27200/41640 -loss: 0.001937
- epoch: 19 - global_step: 27250/41640 -loss: 0.000548
- epoch: 19 - global_step: 27300/41640 -loss: 0.002550
- epoch: 19 - global_step: 27350/41640 -loss: 0.000626
- epoch: 19 - global_step: 27400/41640 -loss: 0.001593
- epoch: 19 - global_step: 27450/41640 -loss: 0.000773
- epoch: 19 - global_step: 27500/41640 -loss: 0.000986
evaluation result: Micro F1 score: 0.93909, Macro F1 score: 0.66645
- epoch: 19 - global_step: 27550/41640 -loss: 0.000272
- epoch: 19 - global_step: 27600/41640 -loss: 0.001385
- epoch: 19 - global_step: 27650/41640 -loss: 0.001204
- epoch: 19 - global_step: 27700/41640 -loss: 0.000232
- epoch: 19 - global_step: 27750/41640 -loss: 0.002729
- epoch: 20 - global_step: 27800/41640 -loss: 0.000507
- epoch: 20 - global_step: 27850/41640 -loss: 0.000825
- epoch: 20 - global_step: 27900/41640 -loss: 0.000443
- epoch: 20 - global_step: 27950/41640 -loss: 0.000120
- epoch: 20 - global_step: 28000/41640 -loss: 0.000187
evaluation result: Micro F1 score: 0.93976, Macro F1 score: 0.63691
- epoch: 20 - global_step: 28050/41640 -loss: 0.000474
- epoch: 20 - global_step: 28100/41640 -loss: 0.002433
- epoch: 20 - global_step: 28150/41640 -loss: 0.000199
- epoch: 20 - global_step: 28200/41640 -loss: 0.002750
- epoch: 20 - global_step: 28250/41640 -loss: 0.000914
- epoch: 20 - global_step: 28300/41640 -loss: 0.001129
- epoch: 20 - global_step: 28350/41640 -loss: 0.000395
- epoch: 20 - global_step: 28400/41640 -loss: 0.001245
- epoch: 20 - global_step: 28450/41640 -loss: 0.000597
- epoch: 20 - global_step: 28500/41640 -loss: 0.000287
evaluation result: Micro F1 score: 0.93925, Macro F1 score: 0.66984
- epoch: 20 - global_step: 28550/41640 -loss: 0.003770
- epoch: 20 - global_step: 28600/41640 -loss: 0.000863
- epoch: 20 - global_step: 28650/41640 -loss: 0.002729
- epoch: 20 - global_step: 28700/41640 -loss: 0.001136
- epoch: 20 - global_step: 28750/41640 -loss: 0.001354
- epoch: 20 - global_step: 28800/41640 -loss: 0.001258
- epoch: 20 - global_step: 28850/41640 -loss: 0.001506
- epoch: 20 - global_step: 28900/41640 -loss: 0.001644
- epoch: 20 - global_step: 28950/41640 -loss: 0.001008
- epoch: 20 - global_step: 29000/41640 -loss: 0.000979
evaluation result: Micro F1 score: 0.94036, Macro F1 score: 0.67626
- epoch: 20 - global_step: 29050/41640 -loss: 0.001828
- epoch: 20 - global_step: 29100/41640 -loss: 0.000957
- epoch: 21 - global_step: 29150/41640 -loss: 0.001045
- epoch: 21 - global_step: 29200/41640 -loss: 0.001230
- epoch: 21 - global_step: 29250/41640 -loss: 0.001424
- epoch: 21 - global_step: 29300/41640 -loss: 0.001117
- epoch: 21 - global_step: 29350/41640 -loss: 0.000698
- epoch: 21 - global_step: 29400/41640 -loss: 0.000238
- epoch: 21 - global_step: 29450/41640 -loss: 0.002784
- epoch: 21 - global_step: 29500/41640 -loss: 0.000378
best F1 performance has been updated: 0.68471 --> 0.69168
evaluation result: Micro F1 score: 0.94011, Macro F1 score: 0.69168
- epoch: 21 - global_step: 29550/41640 -loss: 0.000127
- epoch: 21 - global_step: 29600/41640 -loss: 0.001044
- epoch: 21 - global_step: 29650/41640 -loss: 0.000581
- epoch: 21 - global_step: 29700/41640 -loss: 0.002660
- epoch: 21 - global_step: 29750/41640 -loss: 0.001129
- epoch: 21 - global_step: 29800/41640 -loss: 0.001471
- epoch: 21 - global_step: 29850/41640 -loss: 0.002218
- epoch: 21 - global_step: 29900/41640 -loss: 0.000818
- epoch: 21 - global_step: 29950/41640 -loss: 0.001752
- epoch: 21 - global_step: 30000/41640 -loss: 0.001840
evaluation result: Micro F1 score: 0.93970, Macro F1 score: 0.68079
- epoch: 21 - global_step: 30050/41640 -loss: 0.000172
- epoch: 21 - global_step: 30100/41640 -loss: 0.002039
- epoch: 21 - global_step: 30150/41640 -loss: 0.000166
- epoch: 21 - global_step: 30200/41640 -loss: 0.000940
- epoch: 21 - global_step: 30250/41640 -loss: 0.001387
- epoch: 21 - global_step: 30300/41640 -loss: 0.000285
- epoch: 21 - global_step: 30350/41640 -loss: 0.000844
- epoch: 21 - global_step: 30400/41640 -loss: 0.002102
- epoch: 21 - global_step: 30450/41640 -loss: 0.000638
- epoch: 21 - global_step: 30500/41640 -loss: 0.000960
evaluation result: Micro F1 score: 0.93845, Macro F1 score: 0.66081
- epoch: 22 - global_step: 30550/41640 -loss: 0.000354
- epoch: 22 - global_step: 30600/41640 -loss: 0.000407
- epoch: 22 - global_step: 30650/41640 -loss: 0.000409
- epoch: 22 - global_step: 30700/41640 -loss: 0.000144
- epoch: 22 - global_step: 30750/41640 -loss: 0.001880
- epoch: 22 - global_step: 30800/41640 -loss: 0.000231
- epoch: 22 - global_step: 30850/41640 -loss: 0.001057
- epoch: 22 - global_step: 30900/41640 -loss: 0.001151
- epoch: 22 - global_step: 30950/41640 -loss: 0.000754
- epoch: 22 - global_step: 31000/41640 -loss: 0.000691
evaluation result: Micro F1 score: 0.93690, Macro F1 score: 0.65973
- epoch: 22 - global_step: 31050/41640 -loss: 0.000176
- epoch: 22 - global_step: 31100/41640 -loss: 0.000431
- epoch: 22 - global_step: 31150/41640 -loss: 0.000344
- epoch: 22 - global_step: 31200/41640 -loss: 0.000969
- epoch: 22 - global_step: 31250/41640 -loss: 0.001594
- epoch: 22 - global_step: 31300/41640 -loss: 0.000899
- epoch: 22 - global_step: 31350/41640 -loss: 0.000204
- epoch: 22 - global_step: 31400/41640 -loss: 0.000251
- epoch: 22 - global_step: 31450/41640 -loss: 0.001838
- epoch: 22 - global_step: 31500/41640 -loss: 0.001499
evaluation result: Micro F1 score: 0.93791, Macro F1 score: 0.67383
- epoch: 22 - global_step: 31550/41640 -loss: 0.001138
- epoch: 22 - global_step: 31600/41640 -loss: 0.003066
- epoch: 22 - global_step: 31650/41640 -loss: 0.000209
- epoch: 22 - global_step: 31700/41640 -loss: 0.001024
- epoch: 22 - global_step: 31750/41640 -loss: 0.001403
- epoch: 22 - global_step: 31800/41640 -loss: 0.000926
- epoch: 22 - global_step: 31850/41640 -loss: 0.000271
- epoch: 22 - global_step: 31900/41640 -loss: 0.000173
- epoch: 23 - global_step: 31950/41640 -loss: 0.000903
- epoch: 23 - global_step: 32000/41640 -loss: 0.001724
evaluation result: Micro F1 score: 0.93811, Macro F1 score: 0.64313
- epoch: 23 - global_step: 32050/41640 -loss: 0.000162
- epoch: 23 - global_step: 32100/41640 -loss: 0.001506
- epoch: 23 - global_step: 32150/41640 -loss: 0.000469
- epoch: 23 - global_step: 32200/41640 -loss: 0.000710
- epoch: 23 - global_step: 32250/41640 -loss: 0.000311
- epoch: 23 - global_step: 32300/41640 -loss: 0.001047
- epoch: 23 - global_step: 32350/41640 -loss: 0.000258
- epoch: 23 - global_step: 32400/41640 -loss: 0.001011
- epoch: 23 - global_step: 32450/41640 -loss: 0.000421
- epoch: 23 - global_step: 32500/41640 -loss: 0.000310
best F1 performance has been updated: 0.69168 --> 0.69850
evaluation result: Micro F1 score: 0.93727, Macro F1 score: 0.69850
- epoch: 23 - global_step: 32550/41640 -loss: 0.001920
- epoch: 23 - global_step: 32600/41640 -loss: 0.000460
- epoch: 23 - global_step: 32650/41640 -loss: 0.000777
- epoch: 23 - global_step: 32700/41640 -loss: 0.000376
- epoch: 23 - global_step: 32750/41640 -loss: 0.001459
- epoch: 23 - global_step: 32800/41640 -loss: 0.001244
- epoch: 23 - global_step: 32850/41640 -loss: 0.000974
- epoch: 23 - global_step: 32900/41640 -loss: 0.000799
- epoch: 23 - global_step: 32950/41640 -loss: 0.001474
- epoch: 23 - global_step: 33000/41640 -loss: 0.002838
evaluation result: Micro F1 score: 0.93860, Macro F1 score: 0.67914
- epoch: 23 - global_step: 33050/41640 -loss: 0.000888
- epoch: 23 - global_step: 33100/41640 -loss: 0.000229
- epoch: 23 - global_step: 33150/41640 -loss: 0.004981
- epoch: 23 - global_step: 33200/41640 -loss: 0.001371
- epoch: 23 - global_step: 33250/41640 -loss: 0.002832
- epoch: 23 - global_step: 33300/41640 -loss: 0.000405
- epoch: 24 - global_step: 33350/41640 -loss: 0.000409
- epoch: 24 - global_step: 33400/41640 -loss: 0.000124
- epoch: 24 - global_step: 33450/41640 -loss: 0.001992
- epoch: 24 - global_step: 33500/41640 -loss: 0.000071
evaluation result: Micro F1 score: 0.94253, Macro F1 score: 0.69437
- epoch: 24 - global_step: 33550/41640 -loss: 0.000609
- epoch: 24 - global_step: 33600/41640 -loss: 0.000900
- epoch: 24 - global_step: 33650/41640 -loss: 0.000049
- epoch: 24 - global_step: 33700/41640 -loss: 0.000081
- epoch: 24 - global_step: 33750/41640 -loss: 0.000051
- epoch: 24 - global_step: 33800/41640 -loss: 0.000510
- epoch: 24 - global_step: 33850/41640 -loss: 0.001021
- epoch: 24 - global_step: 33900/41640 -loss: 0.000712
- epoch: 24 - global_step: 33950/41640 -loss: 0.000293
- epoch: 24 - global_step: 34000/41640 -loss: 0.003752
evaluation result: Micro F1 score: 0.93824, Macro F1 score: 0.67786
- epoch: 24 - global_step: 34050/41640 -loss: 0.001226
- epoch: 24 - global_step: 34100/41640 -loss: 0.000722
- epoch: 24 - global_step: 34150/41640 -loss: 0.000384
- epoch: 24 - global_step: 34200/41640 -loss: 0.000417
- epoch: 24 - global_step: 34250/41640 -loss: 0.000241
- epoch: 24 - global_step: 34300/41640 -loss: 0.000162
- epoch: 24 - global_step: 34350/41640 -loss: 0.000270
- epoch: 24 - global_step: 34400/41640 -loss: 0.002009
- epoch: 24 - global_step: 34450/41640 -loss: 0.000839
- epoch: 24 - global_step: 34500/41640 -loss: 0.000270
evaluation result: Micro F1 score: 0.93849, Macro F1 score: 0.68486
- epoch: 24 - global_step: 34550/41640 -loss: 0.001683
- epoch: 24 - global_step: 34600/41640 -loss: 0.000824
- epoch: 24 - global_step: 34650/41640 -loss: 0.001224
- epoch: 25 - global_step: 34700/41640 -loss: 0.000622
- epoch: 25 - global_step: 34750/41640 -loss: 0.000299
- epoch: 25 - global_step: 34800/41640 -loss: 0.000088
- epoch: 25 - global_step: 34850/41640 -loss: 0.000580
- epoch: 25 - global_step: 34900/41640 -loss: 0.000987
- epoch: 25 - global_step: 34950/41640 -loss: 0.000841
- epoch: 25 - global_step: 35000/41640 -loss: 0.000395
evaluation result: Micro F1 score: 0.93968, Macro F1 score: 0.66679
- epoch: 25 - global_step: 35050/41640 -loss: 0.000836
- epoch: 25 - global_step: 35100/41640 -loss: 0.000455
- epoch: 25 - global_step: 35150/41640 -loss: 0.002240
- epoch: 25 - global_step: 35200/41640 -loss: 0.000259
- epoch: 25 - global_step: 35250/41640 -loss: 0.001022
- epoch: 25 - global_step: 35300/41640 -loss: 0.000319
- epoch: 25 - global_step: 35350/41640 -loss: 0.000397
- epoch: 25 - global_step: 35400/41640 -loss: 0.000121
- epoch: 25 - global_step: 35450/41640 -loss: 0.000232
- epoch: 25 - global_step: 35500/41640 -loss: 0.000796
evaluation result: Micro F1 score: 0.94095, Macro F1 score: 0.68759
- epoch: 25 - global_step: 35550/41640 -loss: 0.000142
- epoch: 25 - global_step: 35600/41640 -loss: 0.000312
- epoch: 25 - global_step: 35650/41640 -loss: 0.000483
- epoch: 25 - global_step: 35700/41640 -loss: 0.001362
- epoch: 25 - global_step: 35750/41640 -loss: 0.000891
- epoch: 25 - global_step: 35800/41640 -loss: 0.000455
- epoch: 25 - global_step: 35850/41640 -loss: 0.004052
- epoch: 25 - global_step: 35900/41640 -loss: 0.000110
- epoch: 25 - global_step: 35950/41640 -loss: 0.002845
- epoch: 25 - global_step: 36000/41640 -loss: 0.000424
evaluation result: Micro F1 score: 0.93957, Macro F1 score: 0.68174
- epoch: 25 - global_step: 36050/41640 -loss: 0.000100
- epoch: 26 - global_step: 36100/41640 -loss: 0.000658
- epoch: 26 - global_step: 36150/41640 -loss: 0.000980
- epoch: 26 - global_step: 36200/41640 -loss: 0.000549
- epoch: 26 - global_step: 36250/41640 -loss: 0.000705
- epoch: 26 - global_step: 36300/41640 -loss: 0.001073
- epoch: 26 - global_step: 36350/41640 -loss: 0.001867
- epoch: 26 - global_step: 36400/41640 -loss: 0.000186
- epoch: 26 - global_step: 36450/41640 -loss: 0.000648
- epoch: 26 - global_step: 36500/41640 -loss: 0.001111
evaluation result: Micro F1 score: 0.94149, Macro F1 score: 0.68945
- epoch: 26 - global_step: 36550/41640 -loss: 0.001419
- epoch: 26 - global_step: 36600/41640 -loss: 0.000877
- epoch: 26 - global_step: 36650/41640 -loss: 0.000306
- epoch: 26 - global_step: 36700/41640 -loss: 0.001394
- epoch: 26 - global_step: 36750/41640 -loss: 0.000098
- epoch: 26 - global_step: 36800/41640 -loss: 0.000264
- epoch: 26 - global_step: 36850/41640 -loss: 0.000159
- epoch: 26 - global_step: 36900/41640 -loss: 0.000489
- epoch: 26 - global_step: 36950/41640 -loss: 0.000121
- epoch: 26 - global_step: 37000/41640 -loss: 0.000655
evaluation result: Micro F1 score: 0.93932, Macro F1 score: 0.69555
- epoch: 26 - global_step: 37050/41640 -loss: 0.001580
- epoch: 26 - global_step: 37100/41640 -loss: 0.000311
- epoch: 26 - global_step: 37150/41640 -loss: 0.001398
- epoch: 26 - global_step: 37200/41640 -loss: 0.000340
- epoch: 26 - global_step: 37250/41640 -loss: 0.000344
- epoch: 26 - global_step: 37300/41640 -loss: 0.000368
- epoch: 26 - global_step: 37350/41640 -loss: 0.000829
- epoch: 26 - global_step: 37400/41640 -loss: 0.001498
- epoch: 26 - global_step: 37450/41640 -loss: 0.000470
- epoch: 27 - global_step: 37500/41640 -loss: 0.000638
evaluation result: Micro F1 score: 0.93688, Macro F1 score: 0.65524
- epoch: 27 - global_step: 37550/41640 -loss: 0.000124
- epoch: 27 - global_step: 37600/41640 -loss: 0.000530
- epoch: 27 - global_step: 37650/41640 -loss: 0.000613
- epoch: 27 - global_step: 37700/41640 -loss: 0.000552
- epoch: 27 - global_step: 37750/41640 -loss: 0.000599
- epoch: 27 - global_step: 37800/41640 -loss: 0.001587
- epoch: 27 - global_step: 37850/41640 -loss: 0.000621
- epoch: 27 - global_step: 37900/41640 -loss: 0.000210
- epoch: 27 - global_step: 37950/41640 -loss: 0.000243
- epoch: 27 - global_step: 38000/41640 -loss: 0.000990
evaluation result: Micro F1 score: 0.93605, Macro F1 score: 0.67147
- epoch: 27 - global_step: 38050/41640 -loss: 0.000772
- epoch: 27 - global_step: 38100/41640 -loss: 0.000382
- epoch: 27 - global_step: 38150/41640 -loss: 0.001243
- epoch: 27 - global_step: 38200/41640 -loss: 0.000030
- epoch: 27 - global_step: 38250/41640 -loss: 0.000200
- epoch: 27 - global_step: 38300/41640 -loss: 0.000880
- epoch: 27 - global_step: 38350/41640 -loss: 0.000596
- epoch: 27 - global_step: 38400/41640 -loss: 0.000368
- epoch: 27 - global_step: 38450/41640 -loss: 0.000632
- epoch: 27 - global_step: 38500/41640 -loss: 0.001241
evaluation result: Micro F1 score: 0.94054, Macro F1 score: 0.67754
- epoch: 27 - global_step: 38550/41640 -loss: 0.000672
- epoch: 27 - global_step: 38600/41640 -loss: 0.000644
- epoch: 27 - global_step: 38650/41640 -loss: 0.000399
- epoch: 27 - global_step: 38700/41640 -loss: 0.004502
- epoch: 27 - global_step: 38750/41640 -loss: 0.000270
- epoch: 27 - global_step: 38800/41640 -loss: 0.001166
- epoch: 27 - global_step: 38850/41640 -loss: 0.000367
- epoch: 28 - global_step: 38900/41640 -loss: 0.000126
- epoch: 28 - global_step: 38950/41640 -loss: 0.000574
- epoch: 28 - global_step: 39000/41640 -loss: 0.000583
evaluation result: Micro F1 score: 0.93968, Macro F1 score: 0.68691
- epoch: 28 - global_step: 39050/41640 -loss: 0.000998
- epoch: 28 - global_step: 39100/41640 -loss: 0.000239
- epoch: 28 - global_step: 39150/41640 -loss: 0.001563
- epoch: 28 - global_step: 39200/41640 -loss: 0.001919
- epoch: 28 - global_step: 39250/41640 -loss: 0.003335
- epoch: 28 - global_step: 39300/41640 -loss: 0.000534
- epoch: 28 - global_step: 39350/41640 -loss: 0.000973
- epoch: 28 - global_step: 39400/41640 -loss: 0.000091
- epoch: 28 - global_step: 39450/41640 -loss: 0.000538
- epoch: 28 - global_step: 39500/41640 -loss: 0.001029
evaluation result: Micro F1 score: 0.94000, Macro F1 score: 0.69264
- epoch: 28 - global_step: 39550/41640 -loss: 0.000615
- epoch: 28 - global_step: 39600/41640 -loss: 0.000255
- epoch: 28 - global_step: 39650/41640 -loss: 0.000084
- epoch: 28 - global_step: 39700/41640 -loss: 0.002022
- epoch: 28 - global_step: 39750/41640 -loss: 0.001548
- epoch: 28 - global_step: 39800/41640 -loss: 0.000055
- epoch: 28 - global_step: 39850/41640 -loss: 0.000623
- epoch: 28 - global_step: 39900/41640 -loss: 0.000657
- epoch: 28 - global_step: 39950/41640 -loss: 0.000942
- epoch: 28 - global_step: 40000/41640 -loss: 0.000912
evaluation result: Micro F1 score: 0.93983, Macro F1 score: 0.69614
- epoch: 28 - global_step: 40050/41640 -loss: 0.000202
- epoch: 28 - global_step: 40100/41640 -loss: 0.001253
- epoch: 28 - global_step: 40150/41640 -loss: 0.000364
- epoch: 28 - global_step: 40200/41640 -loss: 0.000228
- epoch: 28 - global_step: 40250/41640 -loss: 0.001285
- epoch: 29 - global_step: 40300/41640 -loss: 0.001045
- epoch: 29 - global_step: 40350/41640 -loss: 0.000787
- epoch: 29 - global_step: 40400/41640 -loss: 0.000431
- epoch: 29 - global_step: 40450/41640 -loss: 0.002730
- epoch: 29 - global_step: 40500/41640 -loss: 0.001002
evaluation result: Micro F1 score: 0.93837, Macro F1 score: 0.68654
- epoch: 29 - global_step: 40550/41640 -loss: 0.000819
- epoch: 29 - global_step: 40600/41640 -loss: 0.000076
- epoch: 29 - global_step: 40650/41640 -loss: 0.000310
- epoch: 29 - global_step: 40700/41640 -loss: 0.001397
- epoch: 29 - global_step: 40750/41640 -loss: 0.001286
- epoch: 29 - global_step: 40800/41640 -loss: 0.000244
- epoch: 29 - global_step: 40850/41640 -loss: 0.000111
- epoch: 29 - global_step: 40900/41640 -loss: 0.001625
- epoch: 29 - global_step: 40950/41640 -loss: 0.000732
- epoch: 29 - global_step: 41000/41640 -loss: 0.000385
evaluation result: Micro F1 score: 0.93945, Macro F1 score: 0.69593
- epoch: 29 - global_step: 41050/41640 -loss: 0.000753
- epoch: 29 - global_step: 41100/41640 -loss: 0.000711
- epoch: 29 - global_step: 41150/41640 -loss: 0.001398
- epoch: 29 - global_step: 41200/41640 -loss: 0.000396
- epoch: 29 - global_step: 41250/41640 -loss: 0.000240
- epoch: 29 - global_step: 41300/41640 -loss: 0.000833
- epoch: 29 - global_step: 41350/41640 -loss: 0.001922
- epoch: 29 - global_step: 41400/41640 -loss: 0.000889
- epoch: 29 - global_step: 41450/41640 -loss: 0.001036
- epoch: 29 - global_step: 41500/41640 -loss: 0.000438
evaluation result: Micro F1 score: 0.94137, Macro F1 score: 0.68331
- epoch: 29 - global_step: 41550/41640 -loss: 0.000133
- epoch: 29 - global_step: 41600/41640 -loss: 0.000192
保存训练过程中的损失train_loss_record和在验证集上的得分train_socre_record，并可视化训练过程，代码实现如下。
import matplotlib.pyplot as plt
def plot_training_loss(train_loss_record, fig_name, fig_size=(8, 6), sample_step=10, loss_legend_loc="lower left", acc_legend_loc="lower left"):
plt.figure(figsize=fig_size)
train_steps = [x[0] for x in train_loss_record][::sample_step]
train_losses = [x[1] for x in train_loss_record][::sample_step]
plt.plot(train_steps, train_losses, color='#e4007f', label="Train Loss")
#绘制坐标轴和图例
plt.ylabel("Loss", fontsize='large')
plt.xlabel("Step", fontsize='large')
plt.legend(loc=loss_legend_loc, fontsize='x-large')
plt.savefig(fig_name)
plt.show()
def plot_training_f1(train_score_record, fig_name, fig_size=(8, 6), sample_step=10, loss_legend_loc="lower left", acc_legend_loc="lower left"):
plt.figure(figsize=fig_size)
train_steps=[x[0] for x in train_score_record]
train_losses = [x[1] for x in train_score_record]
plt.plot(train_steps, train_losses, color='#e4007f', label="Dev Macro F1")
# 绘制坐标轴和图例
plt.ylabel("Macro F1", fontsize='large')
plt.xlabel("Step", fontsize='large')
plt.legend(loc=loss_legend_loc, fontsize='x-large')
plt.savefig(fig_name)
plt.show()
plot_training_loss(train_loss_record, 'loss.jpg')
plot_training_f1(train_score_record, 'f1.jpg')
从输出结果看，随着训练的进行训练集上的损失函数不断下降，然后收敛，数值趋向于0，同时在验证集上的准确率得分起初不断升高，在模型收敛后逐步平稳。
7.模型评估
使用测试集对训练过程中表现最好的模型进行评价，以验证模型训练效果。代码实现如下。
# 加载训练好的模型进行评估。重新实例化一个模型，然后将训练好的模型参数加载到新模型
saved_state = paddle.load("./checkpoints/best.pdparams")
model = AutoModelForSequenceClassification.from_pretrained("ernie-3.0-medium-zh", num_classes=158)
model.load_dict(saved_state)
# 评估模型
micro_f1_score, macro_f1_score = evaluate(model, test_loader, metric)
print(f"test dataset result: Micro F1 score: {micro_f1_score:.5f}, Macro F1 score: {macro_f1_score:.5f}")
[2022-10-19 13:03:54,897] [ INFO] - We are using <class 'paddlenlp.transformers.ernie.modeling.ErnieForSequenceClassification'> to load 'ernie-3.0-medium-zh'.
[2022-10-19 13:03:54,900] [ INFO] - Already cached /home/aistudio/.paddlenlp/models/ernie-3.0-medium-zh/ernie_3.0_medium_zh.pdparams
test dataset result: Micro F1 score: 0.93640, Macro F1 score: 0.67577
8.模型预测
输入待预测文本，通过模型推理验证模型训练效果，代码实现如下。
def infer(model, text):
model.eval()
# 数据处理
encoded_inputs = tokenizer(text=text, max_seq_len=128)
# 构造输入模型的数据
input_ids = paddle.to_tensor(encoded_inputs["input_ids"]).unsqueeze(0)
token_type_ids = paddle.to_tensor(encoded_inputs["token_type_ids"]).unsqueeze(0)
# 计算模型输出分类的对数几率
logits = model(input_ids, token_type_ids)
probs = F.sigmoid(logits).numpy()
# 解析出分数最大的标签
labels = []
for i, p in enumerate(probs[0]):
if p > 0.5:
labels.append(label_map[i])
print("Text: ", text)
print("Label: ", ','.join(labels))
text = "你好，给我推荐一个评分是5分，价格在100-200元的酒店"
infer(model, text)
Text: 你好，给我推荐一个评分是5分，价格在100-200元的酒店
Label: General+greet+none+none,Request+酒店+名称+
加入交流群，一起学习吧
以上实现基于PaddleNLP，开源不易，希望大家多多支持~
如果对您有帮助，记得给PaddleNLP点个小小的Star⭐，收藏起来，不易走丢~
GitHub地址：https://github.com/PaddlePaddle/PaddleNLP
此文章为搬运
原项目链接
学大模型，用大模型上飞桨星河社区！每天8点V100G算力免费领！免费领取ERNIE 4.0 100w Token >>>
更多推荐
0
0- 0
已为社区贡献1438条内容
所有评论(0)