论文复现: DSIN-阿里点击率预测三部曲-3
转自AI Studio,原文链接:论文复现: DSIN-阿里点击率预测三部曲-3 - 飞桨AI Studio论文复现:Deep Session Interest Network for Click-Through Rate Prediction一、简介Deep Session Interest Network for Click-Through Rate Prediction 是点击率预测问题的一
转自AI Studio,原文链接:
论文复现: DSIN-阿里点击率预测三部曲-3 - 飞桨AI Studio
论文复现:Deep Session Interest Network for Click-Through Rate Prediction
一、简介
Deep Session Interest Network for Click-Through Rate Prediction 是点击率预测问题的一篇经典论文,该论文的先前工作有大家非常熟悉的DIN,DIEN。都是关注于用户的兴趣,对用户的历史会话行为建模出用户的兴趣表示。DSIN模型观察到了,用户在 session 中的兴趣是高度相近的,但是在不同的 session 中的兴趣是不同的,如下图所示:
根据上述观察,DSIN将用户的历史交互数据划分成了一个个 session , 然后再通过自注意力和双向LSTM对用户的 session 兴趣进行建模。模型框架与DIN,DIEN类似,如下图所示:
论文连接:Deep Session Interest Network for Click-Through Rate Prediction
二、复现精度
基于paddlepaddle深度学习框架,对文献算法进行复现后,本项目达到的测试精度,如下表所示。
模型 | auc | batch_size | epoch_num | Time of each epoch |
---|---|---|---|---|
DSIN | 0.6356 | 4096 | 1 | 约10分钟 |
参数设置可详见config_bigdata.yaml文件。
三、数据集
本项目所使用的数据集Ali_Display_Ad_Click是由阿里所提供的一个淘宝展示广告点击率预估数据集。
1、原始数据集介绍
- 原始样本骨架raw_sample:淘宝网站中随机抽样了114万用户8天内的广告展示/点击日志(2600万条记录),构成原始的样本骨架
- user:脱敏过的用户ID;
- adgroup_id:脱敏过的广告单元ID;
- time_stamp:时间戳;
- pid:资源位;
- nonclk:为1代表没有点击;为0代表点击;
- clk:为0代表没有点击;为1代表点击;
user,time_stamp,adgroup_id,pid,nonclk,clk
581738,1494137644,1,430548_1007,1,0
- 广告基本信息表ad_feature:本数据集涵盖了raw_sample中全部广告的基本信息
- adgroup_id:脱敏过的广告ID;
- cate_id:脱敏过的商品类目ID;
- campaign_id:脱敏过的广告计划ID;
- customer: 脱敏过的广告主ID;
- brand:脱敏过的品牌ID;
- price: 宝贝的价格
adgroup_id,cate_id,campaign_id,customer,brand,price
63133,6406,83237,1,95471,170.0
- 用户基本信息表user_profile:本数据集涵盖了raw_sample中全部用户的基本信息
- userid:脱敏过的用户ID;
- cms_segid:微群ID;
- cms_group_id:cms_group_id;
- final_gender_code:性别 1:男,2:女;
- age_level:年龄层次; 1234
- pvalue_level:消费档次,1:低档,2:中档,3:高档;
- shopping_level:购物深度,1:浅层用户,2:中度用户,3:深度用户
- occupation:是否大学生 ,1:是,0:否
- new_user_class_level:城市层级
userid,cms_segid,cms_group_id,final_gender_code,age_level,pvalue_level,shopping_level,occupation,new_user_class_level
234,0,5,2,5,,3,0,3
- 用户的行为日志behavior_log:本数据集涵盖了raw_sample中全部用户22天内的购物行为
- user:脱敏过的用户ID;
- time_stamp:时间戳;
- btag:行为类型, 包括以下四种:(pv:浏览),(cart:加入购物车),(fav:喜欢),(buy:购买)
- cate:脱敏过的商品类目id;
- brand: 脱敏过的品牌id;
user,time_stamp,btag,cate,brand
558157,1493741625,pv,6250,91286
预处理数据集介绍
对原始数据集中的四个文件,参考原论文的数据预处理过程对数据进行处理,形成满足DSIN论文条件且可以被reader直接读取的数据集。 数据集共有八个pkl文件,训练集和测试集各自拥有四个,以训练集为例,这四个文件为train_feat_input.pkl、train_sess_input、train_sess_length和train_label.pkl。各自存储了按0.25的采样比进行采样后的user及item特征输入,用户会话特征输入、用户会话长度和标签数据。
四、环境依赖
-
硬件:
- x86 cpu
- NVIDIA GPU
-
框架:
- PaddlePaddle = 2.2.2
- Python = 3.7
-
其他依赖项:
- PaddleRec
五、快速开始
1、克隆PaddleRec
In [1]
#clone PaddleRec
import os
!ls /home/aistudio/data/
!ls work/
!python --version
!pip list | grep paddlepaddle
if not os.path.isdir('work/PaddleRec'):
!cd work && git clone https://gitee.com/paddlepaddle/PaddleRec.git
data131207 PaddleRec Python 3.7.4 paddlepaddle-gpu 2.2.2.post101
2、解压数据集并移动到PaddleRec的datasets里
In [2]
#解压数据集
!tar -zxvf data/data131207/model_input.tar.gz
!mkdir '/home/aistudio/work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/'
!mkdir '/home/aistudio/work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/'
!mkdir '/home/aistudio/work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/'
!mv model_input/test_feat_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/
!mv model_input/test_label.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/
!mv model_input/test_sess_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/
!mv model_input/test_session_length.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/
!mv model_input/train_feat_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/
!mv model_input/train_label.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/
!mv model_input/train_sess_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/
!mv model_input/train_session_length.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/
model_input/ model_input/test_session_length.pkl model_input/test_sess_input.pkl model_input/train_sess_input.pkl model_input/train_feat_input.pkl model_input/test_feat_input.pkl model_input/test_label.pkl model_input/train_label.pkl model_input/train_session_length.pkl
3、写入模型相关代码
In [3]
!mkdir '/home/aistudio/work/PaddleRec/models/rank/dsin'
%cd '/home/aistudio/work/PaddleRec/models/rank/dsin'
/home/aistudio/work/PaddleRec/models/rank/dsin
In [4]
%%writefile net.py
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import paddle
import paddle.nn as nn
import paddle.nn.functional as F
import math
import numpy as np
from sequence_layers import PositionalEncoder, AttentionSequencePoolingLayer, MLP
class DSIN_layer(nn.Layer):
def __init__(self, user_size, adgroup_size, pid_size, cms_segid_size, cms_group_size,
final_gender_size, age_level_size, pvalue_level_size, shopping_level_size,
occupation_size, new_user_class_level_size, campaign_size,customer_size, cate_size, brand_size, # above is all sparse feat size
sparse_embed_size = 4, att_embedding_size = 8, sess_count = 5, sess_max_length = 10, l2_reg_embedding=1e-6):
super().__init__()
# feature size
self.user_size = user_size
self.adgroup_size = adgroup_size
self.pid_size = pid_size
self.cms_segid_size = cms_segid_size
self.cms_group_size = cms_group_size
self.final_gender_size = final_gender_size
self.age_level_size = age_level_size
self.pvalue_level_size = pvalue_level_size
self.shopping_level_size = shopping_level_size
self.occupation_size = occupation_size
self.new_user_class_level_size = new_user_class_level_size
self.campaign_size = campaign_size
self.customer_size = customer_size
self.cate_size = cate_size
self.brand_size = brand_size
# sparse embed size
self.sparse_embed_size = sparse_embed_size
# transform attention embed size
self.att_embedding_size = att_embedding_size
# hyper_parameters
self.sess_count = 5
self.sess_max_length = 10
# sparse embedding layer
self.userid_embeddings_var = paddle.nn.Embedding(
self.user_size,
self.sparse_embed_size,
sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.adgroup_embeddings_var = paddle.nn.Embedding(
self.adgroup_size,
self.sparse_embed_size,
sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.pid_embeddings_var = paddle.nn.Embedding(
self.pid_size,
self.sparse_embed_size,
#sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.cmsid_embeddings_var = paddle.nn.Embedding(
self.cms_segid_size,
self.sparse_embed_size,
#sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.cmsgroup_embeddings_var = paddle.nn.Embedding(
self.cms_group_size,
self.sparse_embed_size,
#sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.gender_embeddings_var = paddle.nn.Embedding(
self.final_gender_size,
self.sparse_embed_size,
#sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.age_embeddings_var = paddle.nn.Embedding(
self.age_level_size,
self.sparse_embed_size,
#sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.pvalue_embeddings_var = paddle.nn.Embedding(
self.pvalue_level_size,
self.sparse_embed_size,
#sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.shopping_embeddings_var = paddle.nn.Embedding(
self.shopping_level_size,
self.sparse_embed_size,
#sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.occupation_embeddings_var = paddle.nn.Embedding(
self.occupation_size,
self.sparse_embed_size,
#sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.new_user_class_level_embeddings_var = paddle.nn.Embedding(
self.new_user_class_level_size,
self.sparse_embed_size,
#sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.campaign_embeddings_var = paddle.nn.Embedding(
self.campaign_size,
self.sparse_embed_size,
sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.customer_embeddings_var = paddle.nn.Embedding(
self.customer_size,
self.sparse_embed_size,
sparse=True,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.cate_embeddings_var = paddle.nn.Embedding(
self.cate_size,
self.sparse_embed_size,
sparse=True,
padding_idx=0,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
self.brand_embeddings_var = paddle.nn.Embedding(
self.brand_size,
self.sparse_embed_size,
sparse=True,
padding_idx=0,
weight_attr=paddle.ParamAttr(
regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))
# sess interest extractor layer
self.position_encoding = PositionalEncoder(2*self.sparse_embed_size)
self.transform = nn.TransformerEncoderLayer(
d_model = self.att_embedding_size,
nhead = 8,
dim_feedforward = 64,
weight_attr = self._get_weight_attr(),
bias_attr= False,
dropout = 0.0)
# sess interest interacting layer
self.bilstm = nn.LSTM(2*self.sparse_embed_size, 2*self.sparse_embed_size, num_layers = 2, direction='bidirectional')
# sess interest activating layer
self.transform_actpool = AttentionSequencePoolingLayer(weight_normalization=True, name='transform')
self.lstm_actpool = AttentionSequencePoolingLayer(weight_normalization=True, name='lstm')
# MLP moudle
self.mlp = MLP(mlp_hidden_units=[77, 200, 80])
def _get_weight_attr(self):
return paddle.ParamAttr(initializer=nn.initializer.TruncatedNormal(std=0.05))
def forward(self, inputs):
'''
inputs : tulpe, (sparse_input, dense_input, sess_input, sess_length)
sparse_input: (N, 15)
dense_input: (N,)
sess_input:(N, 10, 10)
sess_length: (N,)
'''
sparse_input, dense_input, sess_input, sess_length = inputs
#assert(type(sess_length) == paddle.Tensor), f"At Attention SequencePoolingLayer expected inputs[2]'s type is paddle.Tensor, but got {type(sess_length)}"
# sparse and dense feature
self.user = sparse_input[:, 0]
self.adgroup = sparse_input[:, 1]
self.pid = sparse_input[:, 2]
self.cmsid = sparse_input[:, 3]
self.cmsgroup = sparse_input[:, 4]
self.gender = sparse_input[:, 5]
self.age = sparse_input[:, 6]
self.pvalue = sparse_input[:, 7]
self.shopping = sparse_input[:, 8]
self.occupation = sparse_input[:, 9]
self.new_user_class = sparse_input[:, 10]
self.campaign = sparse_input[:, 11]
self.customer = sparse_input[:, 12]
self.cate = sparse_input[:, 13]
self.brand = sparse_input[:, 14]
self.price = dense_input.unsqueeze_(-1)
# sparse feature embedding
self.user_embeded = self.userid_embeddings_var(self.user)
self.adgroup_embeded = self.adgroup_embeddings_var(self.adgroup)
self.pid_embeded = self.pid_embeddings_var(self.pid)
self.cmsid_embeded = self.cmsid_embeddings_var(self.cmsid)
self.cmsgroup_embeded = self.cmsgroup_embeddings_var(self.cmsgroup)
self.gender_embeded = self.gender_embeddings_var(self.gender)
self.age_embeded = self.age_embeddings_var(self.age)
self.pvalue_embeded = self.pvalue_embeddings_var(self.pvalue)
self.shopping_embeded = self.shopping_embeddings_var(self.shopping)
self.occupation_embeded = self.occupation_embeddings_var(self.occupation)
self.new_user_class_embeded = self.new_user_class_level_embeddings_var(self.new_user_class)
self.campaign_embeded = self.campaign_embeddings_var(self.campaign)
self.customer_embeded = self.customer_embeddings_var(self.customer)
self.cate_embeded = self.cate_embeddings_var(self.cate)
self.brand_embeded = self.brand_embeddings_var(self.brand)
# concat query embeded
# Note: query feature is cate_embeded and brand_embeded
query_embeded = paddle.concat([self.cate_embeded,self.brand_embeded],-1)
# concat sparse feature embeded
deep_input_embeded = paddle.concat([self.user_embeded, self.adgroup_embeded, self.pid_embeded, self.cmsid_embeded,
self.cmsgroup_embeded, self.gender_embeded, self.age_embeded, self.pvalue_embeded,
self.shopping_embeded, self.occupation_embeded, self.new_user_class_embeded,
self.campaign_embeded, self.customer_embeded, self.cate_embeded, self.brand_embeded], -1)
# sess_interest_division part
#cate_sess_embeded = self.cate_embeddings_var(paddle.to_tensor(sess_input[:, ::2, :]))
#brand_sess_embeded = self.brand_embeddings_var(paddle.to_tensor(sess_input[:, 1::2, :]))
cate_sess_embeded = self.cate_embeddings_var(sess_input[:, ::2, :])
brand_sess_embeded = self.brand_embeddings_var(sess_input[:, 1::2, :])
# tr_input (n,5,10,8)
tr_input = paddle.concat([cate_sess_embeded,brand_sess_embeded],axis=-1)
# sess interest extractor part
lstm_input = []
for i in range(self.sess_count):
tr_sess_input = self.position_encoding( tr_input[:, i, :, :] )
tr_sess_input = self.transform(tr_sess_input)
tr_sess_input = paddle.mean(tr_sess_input, axis=1, keepdim=True)
lstm_input.append(tr_sess_input)
lstm_input = paddle.concat([lstm_input[0], lstm_input[1], lstm_input[2], lstm_input[3], lstm_input[4]], axis=1)
lstm_output, _ = self.bilstm(lstm_input)
lstm_output = (lstm_output[:, :, :2*self.sparse_embed_size] + lstm_output[:, :, 2*self.sparse_embed_size:])/2
# sess interest activating layer
lstm_input = self.transform_actpool([query_embeded, lstm_input, sess_length])
lstm_output = self.lstm_actpool([query_embeded, lstm_output, sess_length])
# concatenate all moudle output
mlp_input = paddle.concat([deep_input_embeded, paddle.nn.Flatten()(lstm_input), paddle.nn.Flatten()(lstm_output), self.price], axis=-1)
out = self.mlp(mlp_input)
return out
Writing net.py
In [5]
%%writefile sequence_layers.py
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import paddle
import paddle.nn as nn
import numpy as np
import copy
import math
class PositionalEncoder(nn.Layer):
def __init__(self, d_model, max_seq_len=50):
#d_model为嵌入维度
super(PositionalEncoder, self).__init__()
self.d_model = d_model
position = np.array([[pos / np.power(10000, 2. * i / self.d_model)
for i in range(self.d_model)]
for pos in range(max_seq_len)])
# Second part, apply the cosine to even columns and sin to odds.
position[:, 0::2] = np.sin(position[:, 0::2]) # dim 2i
position[:, 1::2] = np.cos(position[:, 1::2]) # dim 2i+1
self.position = self.create_parameter(shape=[max_seq_len,self.d_model],
default_initializer=paddle.nn.initializer.Assign(value=position))
def forward(self, x):
x = x*math.sqrt(self.d_model)
seq_len = x.shape[1]
x = x+self.position[:seq_len,:]
return x
class AttentionSequencePoolingLayer(nn.Layer):
def __init__(self, dnn_units=[8, 64, 16], dnn_activation='sigmoid', weight_normalization=False, name=None):
super().__init__()
self.dnn_units = dnn_units
self.dnn_activation = 'sigmoid'
self.weight_normalization = weight_normalization
self.name = name
layer_list = []
#bn_list = []
for i in range(len(dnn_units)-1):
dnn_layer = nn.Linear(
in_features = self.dnn_units[i] if i != 0 else self.dnn_units[i]*4 ,
out_features = self.dnn_units[i+1],
weight_attr= self._weight_init())
self.add_sublayer(self.name + f'linear_{i}', dnn_layer)
layer_list.append(dnn_layer)
#layer_list.append(copy.deepcopy(dnn_layer))
#bn_layer = nn.BatchNorm(50)
#self.add_sublayer(self.name + f'bn_{i}', bn_layer)
#bn_list.append(bn_layer)
#bn_list.append(copy.deepcopy(bn_layer))
#self.bn_layer = nn.LayerList(bn_list)
self.layers = nn.LayerList(layer_list)
self.dnn = nn.Linear(self.dnn_units[-1], 1, weight_attr=self._weight_init())
self.activation = nn.Sigmoid()
self.soft = nn.Softmax()
def _weight_init(self):
return paddle.framework.ParamAttr(initializer=paddle.nn.initializer.XavierNormal())
def forward(self, inputs):
querys, keys, sess_length = inputs
#assert(type(sess_length) == paddle.Tensor), f"At Attention SequencePoolingLayer expected inputs[2]'s type is paddle.Tensor, but got {type(sess_length)}"
keys_length = keys.shape[1]
key_masks = nn.functional.sequence_mask(sess_length, keys_length)
querys = paddle.tile(querys.unsqueeze(1), [1, keys_length, 1])
att_input = paddle.concat([querys, keys, querys-keys, querys*keys], axis=-1)
for i, layer in enumerate(self.layers):
att_input = layer(att_input)
#att_input = self.bn_layer[i](att_input) # BatchNomalization
att_input = self.activation(att_input) # activation
att_score = self.dnn(att_input) # (N, 50, 1)
att_score = paddle.transpose(att_score, [0, 2, 1]) # (N, 1, 50)
if self.weight_normalization:
paddings = paddle.ones_like(att_score) * (-2 ** 32 + 1)
else:
paddings = paddle.zeros_like(att_score)
att_score = paddle.where(key_masks.unsqueeze(1) == 1, att_score, paddings) # key_masks.unsqueeze in order to keep shape same as att_score
att_score = self.soft(att_score)
out = paddle.matmul(att_score, keys)
return out
class MLP(nn.Layer):
def __init__(self, mlp_hidden_units, use_bn=True):
super().__init__()
self.mlp_hidden_units = mlp_hidden_units
self.acitivation = paddle.nn.Sigmoid()
layer_list = []
for i in range(len(mlp_hidden_units)-1):
dnn_layer = nn.Linear(
in_features = self.mlp_hidden_units[i],
out_features = self.mlp_hidden_units[i+1],
weight_attr= self._weight_init())
self.add_sublayer(f'linear_{i}', dnn_layer)
layer_list.append(dnn_layer)
self.layers = nn.LayerList(layer_list)
self.dense = nn.Linear(self.mlp_hidden_units[-1], 1, bias_attr=True, weight_attr= self._weight_init())
self.predict_layer = nn.Sigmoid()
def _weight_init(self):
return paddle.framework.ParamAttr(initializer=paddle.nn.initializer.XavierNormal())
def forward(self, x):
for layer in self.layers:
x = layer(x)
x = self.acitivation(x)
x = self.dense(x)
x = self.predict_layer(x)
return x
Writing sequence_layers.py
In [6]
%%writefile dygraph_model.py
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import paddle
import paddle.nn as nn
import paddle.nn.functional as F
import math
import net
class DygraphModel():
# define model
def create_model(self, config):
user_size = config.get("hyper_parameters.user_size")
cms_segid_size = config.get("hyper_parameters.cms_segid_size")
cms_group_size = config.get("hyper_parameters.cms_group_size")
final_gender_size = config.get(
"hyper_parameters.final_gender_size")
age_level_size = config.get("hyper_parameters.age_level_size")
pvalue_level_size = config.get("hyper_parameters.pvalue_level_size")
shopping_level_size = config.get(
"hyper_parameters.shopping_level_size")
occupation_size = config.get("hyper_parameters.occupation_size")
new_user_class_level_size = config.get(
"hyper_parameters.new_user_class_level_size")
adgroup_size = config.get("hyper_parameters.adgroup_size")
cate_size = config.get("hyper_parameters.cate_size")
campaign_size = config.get("hyper_parameters.campaign_size")
customer_size = config.get("hyper_parameters.customer_size")
brand_size = config.get("hyper_parameters.brand_size")
pid_size = config.get("hyper_parameters.pid_size")
feat_embed_size = config.get(
"hyper_parameters.feat_embed_size")
dsin_model = net.DSIN_layer(
user_size, adgroup_size, pid_size, cms_segid_size, cms_group_size,
final_gender_size, age_level_size, pvalue_level_size, shopping_level_size,
occupation_size, new_user_class_level_size, campaign_size, customer_size,
cate_size, brand_size, sparse_embed_size=feat_embed_size, l2_reg_embedding=1e-6)
return dsin_model
# define loss function by predicts and label
def create_loss(self, pred, label):
return paddle.nn.BCELoss()(pred,label)
# define feeds which convert numpy of batch data to paddle.tensor
def create_feeds(self, batch_data, config):
data, label = (batch_data[0], batch_data[1], batch_data[2], batch_data[3]), batch_data[-1]
#data, label = batch_data[0], batch_data[1]
label = label.reshape([-1,1])
return label, data
# define optimizer
def create_optimizer(self, dy_model, config):
lr = config.get("hyper_parameters.optimizer.learning_rate", 0.001)
optimizer = paddle.optimizer.Adam(
learning_rate=lr, parameters=dy_model.parameters())
return optimizer
# define metrics such as auc/acc
# multi-task need to define multi metric
def create_metrics(self):
metrics_list_name = ["auc"]
auc_metric = paddle.metric.Auc("ROC")
metrics_list = [auc_metric]
return metrics_list, metrics_list_name
# construct train forward phase
def train_forward(self, dy_model, metrics_list, batch_data, config):
label, input_tensor = self.create_feeds(batch_data, config)
pred = dy_model.forward(input_tensor)
# update metrics
predict_2d = paddle.concat(x=[1 - pred, pred], axis=1)
metrics_list[0].update(preds=predict_2d.numpy(), labels=label.numpy())
loss = self.create_loss(pred,paddle.cast(label, "float32"))
print_dict = {'loss': loss}
# print_dict = None
return loss, metrics_list, print_dict
def infer_forward(self, dy_model, metrics_list, batch_data, config):
label, input_tensor = self.create_feeds(batch_data, config)
pred = dy_model.forward(input_tensor)
# update metrics
predict_2d = paddle.concat(x=[1 - pred, pred], axis=1)
metrics_list[0].update(preds=predict_2d.numpy(), labels=label.numpy())
return metrics_list, None
Writing dygraph_model.py
In [7]
%%writefile dsin_reader.py
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from __future__ import print_function
import numpy as np
from paddle.io import IterableDataset
import pandas as pd
sparse_features = ['userid', 'adgroup_id', 'pid', 'cms_segid', 'cms_group_id', 'final_gender_code', 'age_level',
'pvalue_level', 'shopping_level', 'occupation', 'new_user_class_level ', 'campaign_id',
'customer', 'cate_id', 'brand']
dense_features = ['price']
class RecDataset(IterableDataset):
def __init__(self, file_list, config):
super().__init__()
self.file_list = file_list
data_file = [ f.split('/')[-1] for f in file_list]
mode = data_file[0].split('_')[0]
data_dir = file_list[0].split(data_file[0])[0]
assert(mode == 'train' or mode == 'test' or mode == 'sample'), f"mode must be 'train' or 'test', but get '{mode}'"
feat_input = pd.read_pickle(data_dir + mode + '_feat_input.pkl')
self.sess_input = pd.read_pickle(data_dir + mode + '_sess_input.pkl')
self.sess_length = pd.read_pickle(data_dir + mode + '_session_length.pkl')
self.label = pd.read_pickle(data_dir + mode + '_label.pkl')
if str(type(self.label)).split("'")[1] != 'numpy.ndarray':
self.label = self.label.to_numpy()
self.label = self.label.astype('int64')
self.num_samples = self.label.shape[0]
self.sparse_input = feat_input[sparse_features].to_numpy().astype('int64')
self.dense_input = feat_input[dense_features].to_numpy().reshape(-1).astype('float32')
def __iter__(self):
for i in range(self.num_samples):
yield [self.sparse_input[i, :], self.dense_input[i], self.sess_input[i, :, :], self.sess_length[i], self.label[i]]
Writing dsin_reader.py
In [8]
%%writefile config_bigdata.yaml
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
runner:
train_data_dir: "../../../datasets/Ali_Display_Ad_Click_DSIN/big_train/"
train_reader_path: "dsin_reader" # importlib format
use_gpu: True
use_auc: True
train_batch_size: 4096
epochs: 1
print_interval: 50
model_save_path: "output_model_all_dsin"
test_data_dir: "../../../datasets/Ali_Display_Ad_Click_DSIN/big_test/"
infer_reader_path: "dsin_reader" # importlib format
infer_batch_size: 16384 # 2**14
infer_load_path: "output_model_all_dsin"
infer_start_epoch: 0
infer_end_epoch: 1
# hyper parameters of user-defined network
hyper_parameters:
# optimizer config
optimizer:
class: Adam
learning_rate: 0.00235
# user feature size
user_size: 265442
cms_segid_size: 97
cms_group_size: 13
final_gender_size: 2
age_level_size: 7
pvalue_level_size: 4
shopping_level_size: 3
occupation_size: 2
new_user_class_level_size: 5
# item feature size
adgroup_size: 512431
cate_size: 11859 #max value + 1
campaign_size: 309448
customer_size: 195841
brand_size: 362855 #max value + 1
# context feature size
pid_size: 2
# embedding size
feat_embed_size: 4
Writing config_bigdata.yaml
4、利用PaddleRec的trainer以及infer进行模型训练及其测试
In [9]
!python ../../../tools/trainer.py -m config_bigdata.yaml
2022-05-11 19:50:56,823 - INFO - **************common.configs********** 2022-05-11 19:50:56,823 - INFO - use_gpu: True, use_xpu: False, use_visual: False, train_batch_size: 4096, train_data_dir: ../../../datasets/Ali_Display_Ad_Click_DSIN/big_train/, epochs: 1, print_interval: 50, model_save_path: output_model_all_dsin 2022-05-11 19:50:56,823 - INFO - **************common.configs********** W0511 19:50:56.825248 1525 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1 W0511 19:50:56.831076 1525 device_context.cc:465] device: 0, cuDNN Version: 7.6. 2022-05-11 19:51:01,867 - INFO - read data 2022-05-11 19:51:01,867 - INFO - reader path:dsin_reader 2022-05-11 19:51:13,903 - INFO - epoch: 0, batch_id: 0, auc:0.502794, loss:0.85580873, avg_reader_cost: 0.00291 sec, avg_batch_cost: 0.01317 sec, avg_samples: 81.92000, ips: 6220.65504 ins/s 2022-05-11 19:51:33,319 - INFO - epoch: 0, batch_id: 50, auc:0.495701, loss:0.19559237, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38773 sec, avg_samples: 4096.00000, ips: 10564.02249 ins/s 2022-05-11 19:51:52,451 - INFO - epoch: 0, batch_id: 100, auc:0.499694, loss:0.21434923, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38206 sec, avg_samples: 4096.00000, ips: 10720.87298 ins/s 2022-05-11 19:52:10,842 - INFO - epoch: 0, batch_id: 150, auc:0.512509, loss:0.19038938, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.36725 sec, avg_samples: 4096.00000, ips: 11153.31692 ins/s 2022-05-11 19:52:28,755 - INFO - epoch: 0, batch_id: 200, auc:0.530944, loss:0.20696387, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.35769 sec, avg_samples: 4096.00000, ips: 11451.33054 ins/s 2022-05-11 19:52:46,030 - INFO - epoch: 0, batch_id: 250, auc:0.545280, loss:0.18852976, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.34493 sec, avg_samples: 4096.00000, ips: 11874.79419 ins/s 2022-05-11 19:53:03,111 - INFO - epoch: 0, batch_id: 300, auc:0.558348, loss:0.20377612, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34106 sec, avg_samples: 4096.00000, ips: 12009.68762 ins/s 2022-05-11 19:53:20,102 - INFO - epoch: 0, batch_id: 350, auc:0.567205, loss:0.2231454, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33924 sec, avg_samples: 4096.00000, ips: 12073.90980 ins/s 2022-05-11 19:53:36,952 - INFO - epoch: 0, batch_id: 400, auc:0.572662, loss:0.2543741, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33644 sec, avg_samples: 4096.00000, ips: 12174.55680 ins/s 2022-05-11 19:53:54,328 - INFO - epoch: 0, batch_id: 450, auc:0.577503, loss:0.16823483, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34696 sec, avg_samples: 4096.00000, ips: 11805.51984 ins/s 2022-05-11 19:54:13,481 - INFO - epoch: 0, batch_id: 500, auc:0.580811, loss:0.19309358, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38248 sec, avg_samples: 4096.00000, ips: 10709.07133 ins/s 2022-05-11 19:54:32,650 - INFO - epoch: 0, batch_id: 550, auc:0.584353, loss:0.19425544, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38280 sec, avg_samples: 4096.00000, ips: 10700.23452 ins/s 2022-05-11 19:54:51,018 - INFO - epoch: 0, batch_id: 600, auc:0.587535, loss:0.19358435, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.36678 sec, avg_samples: 4096.00000, ips: 11167.49886 ins/s 2022-05-11 19:55:08,682 - INFO - epoch: 0, batch_id: 650, auc:0.590837, loss:0.21790585, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.35272 sec, avg_samples: 4096.00000, ips: 11612.52946 ins/s 2022-05-11 19:55:26,055 - INFO - epoch: 0, batch_id: 700, auc:0.594234, loss:0.19218928, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34689 sec, avg_samples: 4096.00000, ips: 11807.69064 ins/s 2022-05-11 19:55:43,041 - INFO - epoch: 0, batch_id: 750, auc:0.597527, loss:0.20641877, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33916 sec, avg_samples: 4096.00000, ips: 12076.80625 ins/s 2022-05-11 19:55:59,994 - INFO - epoch: 0, batch_id: 800, auc:0.600670, loss:0.22155708, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33848 sec, avg_samples: 4096.00000, ips: 12101.22339 ins/s 2022-05-11 19:56:17,091 - INFO - epoch: 0, batch_id: 850, auc:0.603358, loss:0.19764367, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34137 sec, avg_samples: 4096.00000, ips: 11998.85636 ins/s 2022-05-11 19:56:34,397 - INFO - epoch: 0, batch_id: 900, auc:0.605445, loss:0.18218887, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.34556 sec, avg_samples: 4096.00000, ips: 11853.31707 ins/s 2022-05-11 19:56:53,374 - INFO - epoch: 0, batch_id: 950, auc:0.606719, loss:0.20349224, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.37895 sec, avg_samples: 4096.00000, ips: 10808.89367 ins/s 2022-05-11 19:57:12,244 - INFO - epoch: 0, batch_id: 1000, auc:0.608219, loss:0.18338634, avg_reader_cost: 0.00016 sec, avg_batch_cost: 0.37685 sec, avg_samples: 4096.00000, ips: 10868.97179 ins/s 2022-05-11 19:57:30,490 - INFO - epoch: 0, batch_id: 1050, auc:0.610018, loss:0.18991007, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.36437 sec, avg_samples: 4096.00000, ips: 11241.38734 ins/s 2022-05-11 19:57:48,290 - INFO - epoch: 0, batch_id: 1100, auc:0.611764, loss:0.19425409, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.35542 sec, avg_samples: 4096.00000, ips: 11524.47769 ins/s 2022-05-11 19:58:05,738 - INFO - epoch: 0, batch_id: 1150, auc:0.613360, loss:0.18417387, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34839 sec, avg_samples: 4096.00000, ips: 11756.97841 ins/s 2022-05-11 19:58:22,780 - INFO - epoch: 0, batch_id: 1200, auc:0.615447, loss:0.2374034, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34027 sec, avg_samples: 4096.00000, ips: 12037.41497 ins/s 2022-05-11 19:58:39,730 - INFO - epoch: 0, batch_id: 1250, auc:0.616718, loss:0.21474466, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.33845 sec, avg_samples: 4096.00000, ips: 12102.39913 ins/s 2022-05-11 19:58:56,387 - INFO - epoch: 0, batch_id: 1300, auc:0.618325, loss:0.17899244, avg_reader_cost: 0.00016 sec, avg_batch_cost: 0.33259 sec, avg_samples: 4096.00000, ips: 12315.36361 ins/s 2022-05-11 19:59:13,529 - INFO - epoch: 0, batch_id: 1350, auc:0.619961, loss:0.21630415, avg_reader_cost: 0.00015 sec, avg_batch_cost: 0.34231 sec, avg_samples: 4096.00000, ips: 11965.62220 ins/s 2022-05-11 19:59:14,210 - INFO - epoch: 0 done, auc: 0.620026,loss:0.14849854, epoch time: 480.97 s 2022-05-11 19:59:14,386 - INFO - Already save model in output_model_all_dsin/0
In [10]
!python ../../../tools/infer.py -m config_bigdata.yaml
2022-05-11 19:59:48,026 - INFO - **************common.configs********** 2022-05-11 19:59:48,026 - INFO - use_gpu: True, use_xpu: False, use_visual: False, infer_batch_size: 16384, test_data_dir: ../../../datasets/Ali_Display_Ad_Click_DSIN/big_test/, start_epoch: 0, end_epoch: 1, print_interval: 50, model_load_path: output_model_all_dsin 2022-05-11 19:59:48,026 - INFO - **************common.configs********** W0511 19:59:48.027812 1904 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1 W0511 19:59:48.033318 1904 device_context.cc:465] device: 0, cuDNN Version: 7.6. 2022-05-11 19:59:52,275 - INFO - read data 2022-05-11 19:59:52,276 - INFO - reader path:dsin_reader 2022-05-11 19:59:53,777 - INFO - load model epoch 0 2022-05-11 19:59:53,777 - INFO - start load model from output_model_all_dsin/0 2022-05-11 19:59:54,438 - INFO - epoch: 0, batch_id: 0, auc: 0.628742, avg_reader_cost: 0.00439 sec, avg_batch_cost: 0.01166 sec, avg_samples: 16384.00000, ips: 1239157.77 ins/s 2022-05-11 20:00:02,133 - INFO - epoch: 0 done, auc: 0.635660, epoch time: 8.36 s
六、代码结构与详细说明
6.1 代码结构(在work/PaddleRec/models/rank/dsin目录下)
├── config_bigdata.yaml # 全量数据配置文件
├── net.py # 模型核心组网(动静统一)
├── sequence_layers.py # 模型组网模块
├── dsin_reader.py # 数据读取程序
├── dygraph_model.py # 构建动态图
6.2 参数说明
可以在 config_bigdata.yaml
中设置训练与评估相关参数,具体如下:
参数 | 默认值 | 说明 | 其他 |
---|---|---|---|
--runner.train_data_dir | None | 训练数据路径 | |
--runner.train_reader_path | dsin_reader | 训练数据读取器文件路径 | |
--runner.use_gpu | True | 是否使用gpu | |
--runner.use_auc | 5 | 是否使用auc | |
--runner.train_batch_size | 4096 | 模型训练的batch大小 | |
--runner.epochs | 1 | 训练epochs | |
--runner.print_interval | 50 | trainer训练时每print_interval个batch就打印一次指标 |
评估的相关参数与训练类似就不展开了,且模型的超参数也详细存放在了config_bigdata.yaml
文件中了。
七、复现心得
本项目复现时遇到了许多的问题,分别对应在数据集获取,模型对齐,精度对齐这几方面。且主要因为源代码模块化程度较高,把很多细节都需要去仔细阅读源码才能彻底明白模型框架,论文上的模型框架图只是提供一个大致的思路。
(1)数据集获取: 因为该论文所使用的数据集是需要经过预处理的,而且原始数据集大小为23G+,所以在一开始复现的时候就在数据集这上面费了不少心思,也正是因为这样我学习到了如何去处理这种大数据。通过逐行读取,划分成多个子数据集的方式,将用户的历史行为数据切分成了多个。
(2)模型对齐: 如前面所说的,在一开始复现的时候我就只通过原文中的模型框架尝试去搭建,可想而知结果肯定是不好的。还是需要去仔细阅读原文代码才能很好的与原文模型进行对齐。(可以通过model summary的方式去观察模型框架以及每个layer的输入输出shape。)
(3)精度对齐: 在精度对齐上面,实在是太费心思了。重点是确保数据集一定要与原文一致,模型一定要与原文对齐。达到这两点后基本上就可以实现精度对齐了。
八、模型信息
训练完成后,模型和相关LOG保存在./output_model_all_dsin目录下。
信息 | 说明 |
---|---|
发布者 | lfyzzz |
时间 | 2022.5.11 |
框架版本 | Paddle 2.2.2 |
应用场景 | 推荐系统 |
支持硬件 | GPU、CPU |
更多推荐
所有评论(0)