论文复现： DSIN-阿里点击率预测三部曲-3

转自AI Studio，原文链接：论文复现： DSIN-阿里点击率预测三部曲-3 - 飞桨AI Studio论文复现：Deep Session Interest Network for Click-Through Rate Prediction一、简介Deep Session Interest Network for Click-Through Rate Prediction 是点击率预测问题的一

AI Studio

846人浏览 · 2022-05-19 17:50:36

AI Studio · 2022-05-19 17:50:36 发布

转自AI Studio，原文链接：

论文复现： DSIN-阿里点击率预测三部曲-3 - 飞桨AI Studio

论文复现：Deep Session Interest Network for Click-Through Rate Prediction

一、简介

Deep Session Interest Network for Click-Through Rate Prediction 是点击率预测问题的一篇经典论文，该论文的先前工作有大家非常熟悉的DIN，DIEN。都是关注于用户的兴趣，对用户的历史会话行为建模出用户的兴趣表示。DSIN模型观察到了，用户在 session 中的兴趣是高度相近的，但是在不同的 session 中的兴趣是不同的，如下图所示：

根据上述观察，DSIN将用户的历史交互数据划分成了一个个 session ，然后再通过自注意力和双向LSTM对用户的 session 兴趣进行建模。模型框架与DIN，DIEN类似，如下图所示：

论文连接：Deep Session Interest Network for Click-Through Rate Prediction

二、复现精度

基于paddlepaddle深度学习框架，对文献算法进行复现后，本项目达到的测试精度，如下表所示。

模型	auc	batch_size	epoch_num	Time of each epoch
DSIN	0.6356	4096	1	约10分钟

参数设置可详见config_bigdata.yaml文件。

三、数据集

本项目所使用的数据集Ali_Display_Ad_Click是由阿里所提供的一个淘宝展示广告点击率预估数据集。

1、原始数据集介绍

原始样本骨架raw_sample：淘宝网站中随机抽样了114万用户8天内的广告展示/点击日志（2600万条记录），构成原始的样本骨架

user：脱敏过的用户ID；
adgroup_id：脱敏过的广告单元ID；
time_stamp：时间戳；
pid：资源位；
nonclk：为1代表没有点击；为0代表点击；
clk：为0代表没有点击；为1代表点击；

user,time_stamp,adgroup_id,pid,nonclk,clk
581738,1494137644,1,430548_1007,1,0

广告基本信息表ad_feature：本数据集涵盖了raw_sample中全部广告的基本信息

adgroup_id：脱敏过的广告ID；
cate_id：脱敏过的商品类目ID；
campaign_id：脱敏过的广告计划ID；
customer: 脱敏过的广告主ID；
brand：脱敏过的品牌ID；
price: 宝贝的价格

adgroup_id,cate_id,campaign_id,customer,brand,price
63133,6406,83237,1,95471,170.0

用户基本信息表user_profile：本数据集涵盖了raw_sample中全部用户的基本信息

userid：脱敏过的用户ID；
cms_segid：微群ID；
cms_group_id：cms_group_id；
final_gender_code：性别 1:男,2:女；
age_level：年龄层次； 1234
pvalue_level：消费档次，1:低档，2:中档，3:高档；
shopping_level：购物深度，1:浅层用户,2:中度用户,3:深度用户
occupation：是否大学生，1:是,0:否
new_user_class_level：城市层级

userid,cms_segid,cms_group_id,final_gender_code,age_level,pvalue_level,shopping_level,occupation,new_user_class_level 
234,0,5,2,5,,3,0,3

用户的行为日志behavior_log：本数据集涵盖了raw_sample中全部用户22天内的购物行为

user：脱敏过的用户ID；
time_stamp：时间戳；
btag：行为类型, 包括以下四种：(pv:浏览),(cart:加入购物车),(fav:喜欢),(buy:购买)
cate：脱敏过的商品类目id；
brand: 脱敏过的品牌id；

user,time_stamp,btag,cate,brand
558157,1493741625,pv,6250,91286

预处理数据集介绍

对原始数据集中的四个文件，参考原论文的数据预处理过程对数据进行处理，形成满足DSIN论文条件且可以被reader直接读取的数据集。数据集共有八个pkl文件，训练集和测试集各自拥有四个，以训练集为例，这四个文件为train_feat_input.pkl、train_sess_input、train_sess_length和train_label.pkl。各自存储了按0.25的采样比进行采样后的user及item特征输入，用户会话特征输入、用户会话长度和标签数据。

四、环境依赖

硬件：
- x86 cpu
- NVIDIA GPU
框架：
- PaddlePaddle = 2.2.2
- Python = 3.7
其他依赖项：
- PaddleRec

五、快速开始

1、克隆PaddleRec

In [1]

#clone PaddleRec
import os
!ls /home/aistudio/data/
!ls work/
!python --version
!pip list | grep paddlepaddle
if not os.path.isdir('work/PaddleRec'):
    !cd work && git clone https://gitee.com/paddlepaddle/PaddleRec.git

data131207
PaddleRec
Python 3.7.4
paddlepaddle-gpu       2.2.2.post101

2、解压数据集并移动到PaddleRec的datasets里

In [2]

#解压数据集
!tar -zxvf data/data131207/model_input.tar.gz
!mkdir '/home/aistudio/work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/'
!mkdir '/home/aistudio/work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/'
!mkdir '/home/aistudio/work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/'
!mv model_input/test_feat_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/
!mv model_input/test_label.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/
!mv model_input/test_sess_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/
!mv model_input/test_session_length.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_test/
!mv model_input/train_feat_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/
!mv model_input/train_label.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/
!mv model_input/train_sess_input.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/
!mv model_input/train_session_length.pkl work/PaddleRec/datasets/Ali_Display_Ad_Click_DSIN/big_train/

model_input/
model_input/test_session_length.pkl
model_input/test_sess_input.pkl
model_input/train_sess_input.pkl
model_input/train_feat_input.pkl
model_input/test_feat_input.pkl
model_input/test_label.pkl
model_input/train_label.pkl
model_input/train_session_length.pkl

3、写入模型相关代码

In [3]

!mkdir '/home/aistudio/work/PaddleRec/models/rank/dsin'
%cd '/home/aistudio/work/PaddleRec/models/rank/dsin'

/home/aistudio/work/PaddleRec/models/rank/dsin

In [4]

%%writefile net.py
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import paddle
import paddle.nn as nn
import paddle.nn.functional as F
import math
import numpy as np
from sequence_layers import PositionalEncoder, AttentionSequencePoolingLayer, MLP

class DSIN_layer(nn.Layer):
    def __init__(self, user_size, adgroup_size, pid_size, cms_segid_size, cms_group_size,
                 final_gender_size, age_level_size, pvalue_level_size, shopping_level_size,
                 occupation_size, new_user_class_level_size, campaign_size,customer_size, cate_size, brand_size,  # above is all sparse feat size
                 sparse_embed_size = 4, att_embedding_size = 8, sess_count = 5, sess_max_length = 10, l2_reg_embedding=1e-6):
        super().__init__()

        # feature size
        self.user_size = user_size
        self.adgroup_size = adgroup_size   
        self.pid_size = pid_size
        self.cms_segid_size = cms_segid_size
        self.cms_group_size = cms_group_size
        self.final_gender_size = final_gender_size
        self.age_level_size = age_level_size
        self.pvalue_level_size = pvalue_level_size
        self.shopping_level_size = shopping_level_size
        self.occupation_size = occupation_size
        self.new_user_class_level_size = new_user_class_level_size
        self.campaign_size = campaign_size
        self.customer_size = customer_size
        self.cate_size = cate_size
        self.brand_size = brand_size

        # sparse embed size
        self.sparse_embed_size = sparse_embed_size

        # transform attention embed size
        self.att_embedding_size = att_embedding_size

        # hyper_parameters
        self.sess_count = 5
        self.sess_max_length = 10

        # sparse embedding layer
        self.userid_embeddings_var = paddle.nn.Embedding(
            self.user_size,
            self.sparse_embed_size,
            sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.adgroup_embeddings_var = paddle.nn.Embedding(
            self.adgroup_size,
            self.sparse_embed_size,
            sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.pid_embeddings_var = paddle.nn.Embedding(
            self.pid_size,
            self.sparse_embed_size,
            #sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.cmsid_embeddings_var = paddle.nn.Embedding(
            self.cms_segid_size,
            self.sparse_embed_size,
            #sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.cmsgroup_embeddings_var = paddle.nn.Embedding(
            self.cms_group_size,
            self.sparse_embed_size,
            #sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.gender_embeddings_var = paddle.nn.Embedding(
            self.final_gender_size,
            self.sparse_embed_size,
            #sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.age_embeddings_var = paddle.nn.Embedding(
            self.age_level_size,
            self.sparse_embed_size,
            #sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.pvalue_embeddings_var = paddle.nn.Embedding(
            self.pvalue_level_size,
            self.sparse_embed_size,
            #sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.shopping_embeddings_var = paddle.nn.Embedding(
            self.shopping_level_size,
            self.sparse_embed_size,
            #sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.occupation_embeddings_var = paddle.nn.Embedding(
            self.occupation_size,
            self.sparse_embed_size,
            #sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.new_user_class_level_embeddings_var = paddle.nn.Embedding(
            self.new_user_class_level_size,
            self.sparse_embed_size,
            #sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.campaign_embeddings_var = paddle.nn.Embedding(
            self.campaign_size,
            self.sparse_embed_size,
            sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.customer_embeddings_var = paddle.nn.Embedding(
            self.customer_size,
            self.sparse_embed_size,
            sparse=True,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.cate_embeddings_var = paddle.nn.Embedding(
            self.cate_size,
            self.sparse_embed_size,
            sparse=True,
            padding_idx=0,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        self.brand_embeddings_var = paddle.nn.Embedding(
            self.brand_size,
            self.sparse_embed_size,
            sparse=True,
            padding_idx=0,
            weight_attr=paddle.ParamAttr(
                regularizer=paddle.regularizer.L2Decay(l2_reg_embedding),
                initializer=nn.initializer.Normal(mean=0.0, std=0.0001)))

        # sess interest extractor layer
        self.position_encoding = PositionalEncoder(2*self.sparse_embed_size)
        self.transform = nn.TransformerEncoderLayer(
            d_model = self.att_embedding_size, 
            nhead = 8,
            dim_feedforward = 64,
            weight_attr = self._get_weight_attr(),
            bias_attr= False,
            dropout = 0.0)

        # sess interest interacting layer
        self.bilstm = nn.LSTM(2*self.sparse_embed_size, 2*self.sparse_embed_size, num_layers = 2, direction='bidirectional')

        # sess interest activating layer
        self.transform_actpool = AttentionSequencePoolingLayer(weight_normalization=True, name='transform')
        self.lstm_actpool = AttentionSequencePoolingLayer(weight_normalization=True, name='lstm')

        # MLP moudle
        self.mlp = MLP(mlp_hidden_units=[77, 200, 80])

    def _get_weight_attr(self):
        return paddle.ParamAttr(initializer=nn.initializer.TruncatedNormal(std=0.05))

    def forward(self, inputs):
        '''
        inputs : tulpe, (sparse_input, dense_input, sess_input, sess_length)
            sparse_input: (N, 15)
            dense_input: (N,)
            sess_input：(N, 10, 10)
            sess_length: (N,)
        '''
        sparse_input, dense_input, sess_input, sess_length = inputs
        #assert(type(sess_length) == paddle.Tensor), f"At Attention SequencePoolingLayer expected inputs[2]'s type is paddle.Tensor, but got {type(sess_length)}"

        # sparse and dense feature
        self.user = sparse_input[:, 0]
        self.adgroup = sparse_input[:, 1]
        self.pid = sparse_input[:, 2]
        self.cmsid = sparse_input[:, 3]
        self.cmsgroup = sparse_input[:, 4]
        self.gender = sparse_input[:, 5]
        self.age = sparse_input[:, 6]
        self.pvalue = sparse_input[:, 7]
        self.shopping = sparse_input[:, 8]
        self.occupation = sparse_input[:, 9]
        self.new_user_class = sparse_input[:, 10]
        self.campaign = sparse_input[:, 11]
        self.customer = sparse_input[:, 12]
        self.cate = sparse_input[:, 13]
        self.brand = sparse_input[:, 14]
        self.price = dense_input.unsqueeze_(-1)

        # sparse feature embedding
        self.user_embeded = self.userid_embeddings_var(self.user)
        self.adgroup_embeded = self.adgroup_embeddings_var(self.adgroup)
        self.pid_embeded = self.pid_embeddings_var(self.pid)
        self.cmsid_embeded = self.cmsid_embeddings_var(self.cmsid)
        self.cmsgroup_embeded = self.cmsgroup_embeddings_var(self.cmsgroup)
        self.gender_embeded = self.gender_embeddings_var(self.gender)
        self.age_embeded = self.age_embeddings_var(self.age)
        self.pvalue_embeded = self.pvalue_embeddings_var(self.pvalue)
        self.shopping_embeded = self.shopping_embeddings_var(self.shopping)
        self.occupation_embeded = self.occupation_embeddings_var(self.occupation)
        self.new_user_class_embeded = self.new_user_class_level_embeddings_var(self.new_user_class)
        self.campaign_embeded = self.campaign_embeddings_var(self.campaign)
        self.customer_embeded = self.customer_embeddings_var(self.customer)
        self.cate_embeded = self.cate_embeddings_var(self.cate)
        self.brand_embeded = self.brand_embeddings_var(self.brand)

        # concat query embeded  
        # Note: query feature is cate_embeded and brand_embeded
        query_embeded = paddle.concat([self.cate_embeded,self.brand_embeded],-1)

        # concat sparse feature embeded  
        deep_input_embeded = paddle.concat([self.user_embeded, self.adgroup_embeded, self.pid_embeded, self.cmsid_embeded,
                                    self.cmsgroup_embeded, self.gender_embeded, self.age_embeded, self.pvalue_embeded,
                                    self.shopping_embeded, self.occupation_embeded, self.new_user_class_embeded,
                                    self.campaign_embeded, self.customer_embeded, self.cate_embeded, self.brand_embeded], -1)

        # sess_interest_division part
        #cate_sess_embeded = self.cate_embeddings_var(paddle.to_tensor(sess_input[:, ::2, :]))
        #brand_sess_embeded = self.brand_embeddings_var(paddle.to_tensor(sess_input[:, 1::2, :]))
        cate_sess_embeded = self.cate_embeddings_var(sess_input[:, ::2, :])
        brand_sess_embeded = self.brand_embeddings_var(sess_input[:, 1::2, :])

        # tr_input (n,5,10,8)
        tr_input = paddle.concat([cate_sess_embeded,brand_sess_embeded],axis=-1) 

        # sess interest extractor part
        lstm_input = []
        for i in range(self.sess_count):
            tr_sess_input = self.position_encoding( tr_input[:, i, :, :] )
            tr_sess_input = self.transform(tr_sess_input)
            tr_sess_input = paddle.mean(tr_sess_input, axis=1, keepdim=True)
            lstm_input.append(tr_sess_input)

        lstm_input = paddle.concat([lstm_input[0], lstm_input[1], lstm_input[2], lstm_input[3], lstm_input[4]], axis=1)
        lstm_output, _ = self.bilstm(lstm_input)
        lstm_output = (lstm_output[:, :, :2*self.sparse_embed_size] + lstm_output[:, :, 2*self.sparse_embed_size:])/2

        # sess interest activating layer
        lstm_input = self.transform_actpool([query_embeded, lstm_input, sess_length])
        lstm_output = self.lstm_actpool([query_embeded, lstm_output, sess_length])

        # concatenate all moudle output
        mlp_input = paddle.concat([deep_input_embeded, paddle.nn.Flatten()(lstm_input), paddle.nn.Flatten()(lstm_output), self.price], axis=-1)

        out = self.mlp(mlp_input)
        return out

Writing net.py

In [5]

%%writefile sequence_layers.py
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import paddle
import paddle.nn as nn
import numpy as np
import copy
import math

class PositionalEncoder(nn.Layer):
    def __init__(self, d_model, max_seq_len=50):
        #d_model为嵌入维度
        super(PositionalEncoder, self).__init__()
        self.d_model = d_model

        position = np.array([[pos / np.power(10000, 2. * i / self.d_model)
                            for i in range(self.d_model)]
                            for pos in range(max_seq_len)]) 
        # Second part, apply the cosine to even columns and sin to odds.
        position[:, 0::2] = np.sin(position[:, 0::2])  # dim 2i
        position[:, 1::2] = np.cos(position[:, 1::2])  # dim 2i+1
        self.position = self.create_parameter(shape=[max_seq_len,self.d_model],
                                            default_initializer=paddle.nn.initializer.Assign(value=position))

    def forward(self, x):
        x = x*math.sqrt(self.d_model)
        seq_len = x.shape[1]
        x = x+self.position[:seq_len,:]
        return x

class AttentionSequencePoolingLayer(nn.Layer):
    def __init__(self, dnn_units=[8, 64, 16], dnn_activation='sigmoid', weight_normalization=False, name=None):
        super().__init__()
        self.dnn_units = dnn_units
        self.dnn_activation = 'sigmoid'
        self.weight_normalization = weight_normalization
        self.name = name
        layer_list = []
        #bn_list = []
        for i in range(len(dnn_units)-1):
            dnn_layer = nn.Linear(
                in_features = self.dnn_units[i] if i != 0 else self.dnn_units[i]*4 ,
                out_features = self.dnn_units[i+1],  
                weight_attr= self._weight_init())
            self.add_sublayer(self.name + f'linear_{i}', dnn_layer)
            layer_list.append(dnn_layer)
            #layer_list.append(copy.deepcopy(dnn_layer))
            #bn_layer = nn.BatchNorm(50)
            #self.add_sublayer(self.name + f'bn_{i}', bn_layer)
            #bn_list.append(bn_layer)
            #bn_list.append(copy.deepcopy(bn_layer))
        #self.bn_layer = nn.LayerList(bn_list)
        self.layers = nn.LayerList(layer_list)
        self.dnn = nn.Linear(self.dnn_units[-1], 1, weight_attr=self._weight_init()) 
        self.activation = nn.Sigmoid()
        self.soft = nn.Softmax()
    def _weight_init(self):
        return paddle.framework.ParamAttr(initializer=paddle.nn.initializer.XavierNormal())

    def forward(self, inputs):
        querys, keys, sess_length = inputs
        #assert(type(sess_length) == paddle.Tensor), f"At Attention SequencePoolingLayer expected inputs[2]'s type is paddle.Tensor, but got {type(sess_length)}"
        keys_length = keys.shape[1]
        key_masks = nn.functional.sequence_mask(sess_length, keys_length) 
        querys = paddle.tile(querys.unsqueeze(1), [1, keys_length, 1])
        att_input = paddle.concat([querys, keys, querys-keys, querys*keys], axis=-1)
        for i, layer in enumerate(self.layers):
            att_input = layer(att_input)
            #att_input = self.bn_layer[i](att_input)  # BatchNomalization
            att_input = self.activation(att_input) # activation 
        att_score = self.dnn(att_input)  # (N, 50, 1)
        att_score = paddle.transpose(att_score, [0, 2, 1]) # (N, 1, 50)
        if self.weight_normalization:
            paddings = paddle.ones_like(att_score) * (-2 ** 32 + 1)
        else:
            paddings = paddle.zeros_like(att_score)
        att_score = paddle.where(key_masks.unsqueeze(1) == 1, att_score, paddings)  # key_masks.unsqueeze in order to keep shape same as att_score
        att_score = self.soft(att_score)
        out = paddle.matmul(att_score, keys)
        return out

class MLP(nn.Layer):
    def __init__(self, mlp_hidden_units, use_bn=True):
        super().__init__()
        self.mlp_hidden_units = mlp_hidden_units
        self.acitivation = paddle.nn.Sigmoid()
        layer_list = []
        for i in range(len(mlp_hidden_units)-1):
            dnn_layer = nn.Linear(
                in_features = self.mlp_hidden_units[i],
                out_features = self.mlp_hidden_units[i+1],  
                weight_attr= self._weight_init())
            self.add_sublayer(f'linear_{i}', dnn_layer)
            layer_list.append(dnn_layer)
        self.layers = nn.LayerList(layer_list)
        self.dense = nn.Linear(self.mlp_hidden_units[-1], 1, bias_attr=True, weight_attr= self._weight_init())
        self.predict_layer = nn.Sigmoid()

    def _weight_init(self):
        return paddle.framework.ParamAttr(initializer=paddle.nn.initializer.XavierNormal())

    def forward(self, x):
        for layer in self.layers:
            x = layer(x)
            x = self.acitivation(x)
        x = self.dense(x)
        x = self.predict_layer(x)
        return x

Writing sequence_layers.py

In [6]

%%writefile dygraph_model.py
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.


import paddle
import paddle.nn as nn
import paddle.nn.functional as F
import math

import net


class DygraphModel():
    # define model
    def create_model(self, config):
        user_size = config.get("hyper_parameters.user_size")
        cms_segid_size = config.get("hyper_parameters.cms_segid_size")
        cms_group_size = config.get("hyper_parameters.cms_group_size")
        final_gender_size = config.get(
            "hyper_parameters.final_gender_size")
        age_level_size = config.get("hyper_parameters.age_level_size")
        pvalue_level_size = config.get("hyper_parameters.pvalue_level_size")
        shopping_level_size = config.get(
            "hyper_parameters.shopping_level_size")
        occupation_size = config.get("hyper_parameters.occupation_size")
        new_user_class_level_size = config.get(
            "hyper_parameters.new_user_class_level_size")
        adgroup_size = config.get("hyper_parameters.adgroup_size")
        cate_size = config.get("hyper_parameters.cate_size")
        campaign_size = config.get("hyper_parameters.campaign_size")
        customer_size = config.get("hyper_parameters.customer_size")
        brand_size = config.get("hyper_parameters.brand_size")
        pid_size = config.get("hyper_parameters.pid_size")
        feat_embed_size = config.get(
            "hyper_parameters.feat_embed_size")

        dsin_model = net.DSIN_layer(
            user_size, adgroup_size, pid_size, cms_segid_size, cms_group_size,
            final_gender_size, age_level_size, pvalue_level_size, shopping_level_size,
            occupation_size, new_user_class_level_size, campaign_size, customer_size,
            cate_size, brand_size, sparse_embed_size=feat_embed_size, l2_reg_embedding=1e-6)

        return dsin_model

    # define loss function by predicts and label
    def create_loss(self, pred, label):
        return paddle.nn.BCELoss()(pred,label)

    # define feeds which convert numpy of batch data to paddle.tensor
    def create_feeds(self, batch_data, config):
        data, label = (batch_data[0], batch_data[1], batch_data[2], batch_data[3]), batch_data[-1]
        #data, label = batch_data[0], batch_data[1]
        label = label.reshape([-1,1])
        return label, data

    # define optimizer
    def create_optimizer(self, dy_model, config):
        lr = config.get("hyper_parameters.optimizer.learning_rate", 0.001)
        optimizer = paddle.optimizer.Adam(
            learning_rate=lr, parameters=dy_model.parameters())
        return optimizer

    # define metrics such as auc/acc
    # multi-task need to define multi metric
    def create_metrics(self):
        metrics_list_name = ["auc"]
        auc_metric = paddle.metric.Auc("ROC")
        metrics_list = [auc_metric]
        return metrics_list, metrics_list_name

    # construct train forward phase
    def train_forward(self, dy_model, metrics_list, batch_data, config):
        label, input_tensor = self.create_feeds(batch_data, config)

        pred = dy_model.forward(input_tensor)
        # update metrics
        predict_2d = paddle.concat(x=[1 - pred, pred], axis=1)
        metrics_list[0].update(preds=predict_2d.numpy(), labels=label.numpy())
        loss = self.create_loss(pred,paddle.cast(label, "float32"))
        print_dict = {'loss': loss}
        # print_dict = None
        return loss, metrics_list, print_dict

    def infer_forward(self, dy_model, metrics_list, batch_data, config):
        label, input_tensor = self.create_feeds(batch_data, config)

        pred = dy_model.forward(input_tensor)
        # update metrics
        predict_2d = paddle.concat(x=[1 - pred, pred], axis=1)
        metrics_list[0].update(preds=predict_2d.numpy(), labels=label.numpy())

        return metrics_list, None

Writing dygraph_model.py

In [7]

%%writefile dsin_reader.py
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

from __future__ import print_function
import numpy as np

from paddle.io import IterableDataset
import pandas as pd

sparse_features = ['userid', 'adgroup_id', 'pid', 'cms_segid', 'cms_group_id', 'final_gender_code', 'age_level',
                    'pvalue_level', 'shopping_level', 'occupation', 'new_user_class_level ', 'campaign_id',
                    'customer', 'cate_id', 'brand']

dense_features = ['price']

class RecDataset(IterableDataset):
    def __init__(self, file_list, config):
        super().__init__()
        self.file_list = file_list
        data_file = [ f.split('/')[-1] for f in file_list]
        mode = data_file[0].split('_')[0]
        data_dir = file_list[0].split(data_file[0])[0]
        assert(mode == 'train' or mode == 'test' or mode == 'sample'), f"mode must be 'train' or 'test', but get '{mode}'"
        feat_input = pd.read_pickle(data_dir + mode + '_feat_input.pkl')
        self.sess_input = pd.read_pickle(data_dir + mode + '_sess_input.pkl')
        self.sess_length = pd.read_pickle(data_dir + mode + '_session_length.pkl')
        self.label = pd.read_pickle(data_dir + mode + '_label.pkl')
        if str(type(self.label)).split("'")[1] != 'numpy.ndarray':
            self.label = self.label.to_numpy()
        self.label = self.label.astype('int64')
        self.num_samples = self.label.shape[0]
        self.sparse_input = feat_input[sparse_features].to_numpy().astype('int64')
        self.dense_input = feat_input[dense_features].to_numpy().reshape(-1).astype('float32')

    def __iter__(self):
        for i in range(self.num_samples):
            yield [self.sparse_input[i, :], self.dense_input[i], self.sess_input[i, :, :], self.sess_length[i], self.label[i]]

Writing dsin_reader.py

In [8]

%%writefile config_bigdata.yaml
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

runner:
  train_data_dir: "../../../datasets/Ali_Display_Ad_Click_DSIN/big_train/"
  train_reader_path: "dsin_reader" # importlib format
  use_gpu: True
  use_auc: True
  train_batch_size: 4096
  epochs: 1
  print_interval: 50

  model_save_path: "output_model_all_dsin"
  test_data_dir: "../../../datasets/Ali_Display_Ad_Click_DSIN/big_test/"
  infer_reader_path: "dsin_reader" # importlib format
  infer_batch_size: 16384 # 2**14
  infer_load_path: "output_model_all_dsin"
  infer_start_epoch: 0
  infer_end_epoch: 1

# hyper parameters of user-defined network
hyper_parameters:
  # optimizer config
  optimizer:
    class: Adam
    learning_rate: 0.00235
  # user feature size
  user_size: 265442
  cms_segid_size: 97
  cms_group_size: 13
  final_gender_size: 2
  age_level_size: 7
  pvalue_level_size: 4
  shopping_level_size: 3
  occupation_size: 2
  new_user_class_level_size: 5

  # item feature size
  adgroup_size: 512431
  cate_size: 11859   #max value + 1
  campaign_size: 309448
  customer_size: 195841
  brand_size: 362855  #max value + 1

  # context feature size
  pid_size: 2

  # embedding size
  feat_embed_size: 4

Writing config_bigdata.yaml

4、利用PaddleRec的trainer以及infer进行模型训练及其测试

In [9]

!python ../../../tools/trainer.py -m config_bigdata.yaml

2022-05-11 19:50:56,823 - INFO - **************common.configs**********
2022-05-11 19:50:56,823 - INFO - use_gpu: True, use_xpu: False, use_visual: False, train_batch_size: 4096, train_data_dir: ../../../datasets/Ali_Display_Ad_Click_DSIN/big_train/, epochs: 1, print_interval: 50, model_save_path: output_model_all_dsin
2022-05-11 19:50:56,823 - INFO - **************common.configs**********
W0511 19:50:56.825248  1525 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W0511 19:50:56.831076  1525 device_context.cc:465] device: 0, cuDNN Version: 7.6.
2022-05-11 19:51:01,867 - INFO - read data
2022-05-11 19:51:01,867 - INFO - reader path:dsin_reader
2022-05-11 19:51:13,903 - INFO - epoch: 0, batch_id: 0, auc:0.502794, loss:0.85580873, avg_reader_cost: 0.00291 sec, avg_batch_cost: 0.01317 sec, avg_samples: 81.92000, ips: 6220.65504 ins/s
2022-05-11 19:51:33,319 - INFO - epoch: 0, batch_id: 50, auc:0.495701, loss:0.19559237, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38773 sec, avg_samples: 4096.00000, ips: 10564.02249 ins/s
2022-05-11 19:51:52,451 - INFO - epoch: 0, batch_id: 100, auc:0.499694, loss:0.21434923, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38206 sec, avg_samples: 4096.00000, ips: 10720.87298 ins/s
2022-05-11 19:52:10,842 - INFO - epoch: 0, batch_id: 150, auc:0.512509, loss:0.19038938, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.36725 sec, avg_samples: 4096.00000, ips: 11153.31692 ins/s
2022-05-11 19:52:28,755 - INFO - epoch: 0, batch_id: 200, auc:0.530944, loss:0.20696387, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.35769 sec, avg_samples: 4096.00000, ips: 11451.33054 ins/s
2022-05-11 19:52:46,030 - INFO - epoch: 0, batch_id: 250, auc:0.545280, loss:0.18852976, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.34493 sec, avg_samples: 4096.00000, ips: 11874.79419 ins/s
2022-05-11 19:53:03,111 - INFO - epoch: 0, batch_id: 300, auc:0.558348, loss:0.20377612, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34106 sec, avg_samples: 4096.00000, ips: 12009.68762 ins/s
2022-05-11 19:53:20,102 - INFO - epoch: 0, batch_id: 350, auc:0.567205, loss:0.2231454, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33924 sec, avg_samples: 4096.00000, ips: 12073.90980 ins/s
2022-05-11 19:53:36,952 - INFO - epoch: 0, batch_id: 400, auc:0.572662, loss:0.2543741, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33644 sec, avg_samples: 4096.00000, ips: 12174.55680 ins/s
2022-05-11 19:53:54,328 - INFO - epoch: 0, batch_id: 450, auc:0.577503, loss:0.16823483, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34696 sec, avg_samples: 4096.00000, ips: 11805.51984 ins/s
2022-05-11 19:54:13,481 - INFO - epoch: 0, batch_id: 500, auc:0.580811, loss:0.19309358, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38248 sec, avg_samples: 4096.00000, ips: 10709.07133 ins/s
2022-05-11 19:54:32,650 - INFO - epoch: 0, batch_id: 550, auc:0.584353, loss:0.19425544, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.38280 sec, avg_samples: 4096.00000, ips: 10700.23452 ins/s
2022-05-11 19:54:51,018 - INFO - epoch: 0, batch_id: 600, auc:0.587535, loss:0.19358435, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.36678 sec, avg_samples: 4096.00000, ips: 11167.49886 ins/s
2022-05-11 19:55:08,682 - INFO - epoch: 0, batch_id: 650, auc:0.590837, loss:0.21790585, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.35272 sec, avg_samples: 4096.00000, ips: 11612.52946 ins/s
2022-05-11 19:55:26,055 - INFO - epoch: 0, batch_id: 700, auc:0.594234, loss:0.19218928, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34689 sec, avg_samples: 4096.00000, ips: 11807.69064 ins/s
2022-05-11 19:55:43,041 - INFO - epoch: 0, batch_id: 750, auc:0.597527, loss:0.20641877, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33916 sec, avg_samples: 4096.00000, ips: 12076.80625 ins/s
2022-05-11 19:55:59,994 - INFO - epoch: 0, batch_id: 800, auc:0.600670, loss:0.22155708, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.33848 sec, avg_samples: 4096.00000, ips: 12101.22339 ins/s
2022-05-11 19:56:17,091 - INFO - epoch: 0, batch_id: 850, auc:0.603358, loss:0.19764367, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34137 sec, avg_samples: 4096.00000, ips: 11998.85636 ins/s
2022-05-11 19:56:34,397 - INFO - epoch: 0, batch_id: 900, auc:0.605445, loss:0.18218887, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.34556 sec, avg_samples: 4096.00000, ips: 11853.31707 ins/s
2022-05-11 19:56:53,374 - INFO - epoch: 0, batch_id: 950, auc:0.606719, loss:0.20349224, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.37895 sec, avg_samples: 4096.00000, ips: 10808.89367 ins/s
2022-05-11 19:57:12,244 - INFO - epoch: 0, batch_id: 1000, auc:0.608219, loss:0.18338634, avg_reader_cost: 0.00016 sec, avg_batch_cost: 0.37685 sec, avg_samples: 4096.00000, ips: 10868.97179 ins/s
2022-05-11 19:57:30,490 - INFO - epoch: 0, batch_id: 1050, auc:0.610018, loss:0.18991007, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.36437 sec, avg_samples: 4096.00000, ips: 11241.38734 ins/s
2022-05-11 19:57:48,290 - INFO - epoch: 0, batch_id: 1100, auc:0.611764, loss:0.19425409, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.35542 sec, avg_samples: 4096.00000, ips: 11524.47769 ins/s
2022-05-11 19:58:05,738 - INFO - epoch: 0, batch_id: 1150, auc:0.613360, loss:0.18417387, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34839 sec, avg_samples: 4096.00000, ips: 11756.97841 ins/s
2022-05-11 19:58:22,780 - INFO - epoch: 0, batch_id: 1200, auc:0.615447, loss:0.2374034, avg_reader_cost: 0.00018 sec, avg_batch_cost: 0.34027 sec, avg_samples: 4096.00000, ips: 12037.41497 ins/s
2022-05-11 19:58:39,730 - INFO - epoch: 0, batch_id: 1250, auc:0.616718, loss:0.21474466, avg_reader_cost: 0.00017 sec, avg_batch_cost: 0.33845 sec, avg_samples: 4096.00000, ips: 12102.39913 ins/s
2022-05-11 19:58:56,387 - INFO - epoch: 0, batch_id: 1300, auc:0.618325, loss:0.17899244, avg_reader_cost: 0.00016 sec, avg_batch_cost: 0.33259 sec, avg_samples: 4096.00000, ips: 12315.36361 ins/s
2022-05-11 19:59:13,529 - INFO - epoch: 0, batch_id: 1350, auc:0.619961, loss:0.21630415, avg_reader_cost: 0.00015 sec, avg_batch_cost: 0.34231 sec, avg_samples: 4096.00000, ips: 11965.62220 ins/s
2022-05-11 19:59:14,210 - INFO - epoch: 0 done, auc: 0.620026,loss:0.14849854, epoch time: 480.97 s
2022-05-11 19:59:14,386 - INFO - Already save model in output_model_all_dsin/0

In [10]

!python ../../../tools/infer.py -m config_bigdata.yaml

2022-05-11 19:59:48,026 - INFO - **************common.configs**********
2022-05-11 19:59:48,026 - INFO - use_gpu: True, use_xpu: False, use_visual: False, infer_batch_size: 16384, test_data_dir: ../../../datasets/Ali_Display_Ad_Click_DSIN/big_test/, start_epoch: 0, end_epoch: 1, print_interval: 50, model_load_path: output_model_all_dsin
2022-05-11 19:59:48,026 - INFO - **************common.configs**********
W0511 19:59:48.027812  1904 device_context.cc:447] Please NOTE: device: 0, GPU Compute Capability: 7.0, Driver API Version: 10.1, Runtime API Version: 10.1
W0511 19:59:48.033318  1904 device_context.cc:465] device: 0, cuDNN Version: 7.6.
2022-05-11 19:59:52,275 - INFO - read data
2022-05-11 19:59:52,276 - INFO - reader path:dsin_reader
2022-05-11 19:59:53,777 - INFO - load model epoch 0
2022-05-11 19:59:53,777 - INFO - start load model from output_model_all_dsin/0
2022-05-11 19:59:54,438 - INFO - epoch: 0, batch_id: 0, auc: 0.628742, avg_reader_cost: 0.00439 sec, avg_batch_cost: 0.01166 sec, avg_samples: 16384.00000, ips: 1239157.77 ins/s
2022-05-11 20:00:02,133 - INFO - epoch: 0 done, auc: 0.635660, epoch time: 8.36 s

六、代码结构与详细说明

6.1 代码结构(在work/PaddleRec/models/rank/dsin目录下)

├── config_bigdata.yaml # 全量数据配置文件
├── net.py # 模型核心组网（动静统一）
├── sequence_layers.py # 模型组网模块
├── dsin_reader.py # 数据读取程序
├── dygraph_model.py # 构建动态图

6.2 参数说明

可以在 config_bigdata.yaml 中设置训练与评估相关参数，具体如下：

参数	默认值	说明
--runner.train_data_dir	None	训练数据路径
--runner.train_reader_path	dsin_reader	训练数据读取器文件路径
--runner.use_gpu	True	是否使用gpu
--runner.use_auc	5	是否使用auc
--runner.train_batch_size	4096	模型训练的batch大小
--runner.epochs	1	训练epochs
--runner.print_interval	50	trainer训练时每print_interval个batch就打印一次指标