梗图之王Dalle - 文字生成图片（Text-To-Image）

PaddleNLP新加入了Dalle模型，借助Dalle模型的力量可以直接使用英文描述生成各种脑洞大开的图片，直接调用Dalle模型看看能生成什么有趣例子

生成步骤很简单，仅需要三步，下面开始讲解，go！

注意：运行项目，请至少使用不少32GB显存的GPU镜像

先做准备工作

目前最新版本PaddleNLP-2.3.5，尚未在transformer中添加dalle，需要在github上直接git获取新加入的dallebart代码。
源码地址

1. 从github上copy了dallebart的modeling.py和tokenizer.py文件放入新创建的dalle文件夹

2. 更新PaddleNLP和安装先安装所需的相关依赖

!pip install -U paddlenlp
!pip install emoji
!pip install ftfy
!pip install unidecode
!pip install regex

1 加载模型和tokenizer

PaddleNLP提供了4个版本的DALLE模型,dalle-mini、dalle-mega-v16、dalle-mega-v26、dalle-mega

其中dalle-mega和dalle-mega-v26的预训练模型下载地址是一致，所以准确来说只有3个版本可以使用。

pretrained_resource_files_map = {
    "model_state": {
        "dalle-mini":
        "https://bj.bcebos.com/paddlenlp/models/transformers/dallebart/dalle-mini/model_state.pdparams",
        "dalle-mega-v16":
        "https://bj.bcebos.com/paddlenlp/models/transformers/dallebart/dalle-mega-v16/model_state.pdparams",
        "dalle-mega-v26":
        "https://bj.bcebos.com/paddlenlp/models/transformers/dallebart/dalle-mega-v26/model_state.pdparams",
        "dalle-mega":
        "https://bj.bcebos.com/paddlenlp/models/transformers/dallebart/dalle-mega-v26/model_state.pdparams",
    }
}

dalle-mini约为1.8GB左右，dalle-mega约为8.9GB左右，项目使用mini进行演示，替换成mega可以生成更高质量的图片

import time
import paddle
from dallebart.modeling import DalleBartForImageGeneration
from dallebart.tokenizer import DalleBartTokenizer
from PIL import Image

paddle.seed(0)
paddle.set_device("gpu:0")

# 加载模型和tokenizer
model_name_or_path = "dalle-mini" #dalle-mega
model = DalleBartForImageGeneration.from_pretrained(model_name_or_path)
tokenizer = DalleBartTokenizer.from_pretrained(model_name_or_path)

2 创建一个Text2image的Taskflow

仿类似Taskflow的结构，创建一个Text2image方便演示

# 创建一个Task
class Taskflow():
    def __init__(self, task, model=None, mode=None, device_id=0, num_return_sequences=8, returnTensor=False, saveimg=True, **kwargs):
        self.task = task
        self.model = model
        self.tokenizer = tokenizer
        self.num_return_sequences = num_return_sequences # 返回图片的数量
        self.returnTensor = returnTensor # 是否返回图片Tensor
        self.saveimg = saveimg # 是否保存图片
    
    def task_genration(self,prompt):

        self.model.eval()
        # 可选择的超参数
        top_k = 128
        condition_scale = 10.0
        num_return_sequences = self.num_return_sequences
        
        tokenized_inputs = self.tokenizer(
            prompt,
            return_tensors="pd",
            padding="max_length",
            truncation=True,
            return_attention_mask=True,
            max_length=64,
        )
        
        images = self.model.generate(**tokenized_inputs,
                            top_k=top_k,
                            condition_scale=condition_scale,
                            num_return_sequences=num_return_sequences)
                            
        images = (images.cpu().numpy().clip(0, 1) * 255).astype("uint8")
        images = images.transpose([0, 2, 1, 3, 4]).reshape(-1, images.shape[-3],num_return_sequences * images.shape[-2], images.shape[-1])
        return images[0]
        
    def __call__(self, *inputs):
        """
        The main work function in the taskflow.
        """
        results = self.task_genration(inputs)
        image = Image.fromarray(results)
        image.show()
        if self.saveimg:
            image.save("results/figure_{}.png".format(int(time.time())))
        if self.returnTensor:
            return results

3 调用Dalle生成图片

# 创建文字转换为图像实例
text2image = Taskflow("text_to_image",model,tokenizer,num_return_sequences=8)

例子1 梗图生成

奥特骑车

下方式AI通过珍贵的监控摄像头，记录下了奥特曼骑车上班的真实画面

# 例子1：监控画面拍摄到奥特曼骑自行车
# 输入
prompt = "CCTV footage of Ultraman goes to work by bike"
# 输出
text2image(prompt)

例子2 创建王者荣耀新英雄

# 例子2：通过文字描述创建你的王者荣耀新英雄
# "为王者荣耀设计一个新战士英雄，他使用弯刀作为武器，身后有紫色披风，身穿黑金软甲，头戴帽"
# 输入
prompt = "Designed a new warrior character for Arena of Valor. He uses machetes as weapons, and he wears purple armor, and he has this black and gold cape on the back, and he wears a hat on his head"
# 输出
text2image(prompt)

例子3 插画生成（小王子）

# 例子3：小王子-插画生成
# 输入
prompt = "A little prince with a rose in his hand and a fox were sitting alone on a star looking out into the distance"
# 输出
text2image(prompt)

更多尝试

# 更多尝试,生成卡通头像，家装设计，场景生成，创意时尚单品
# 输入
prompts = [
    "The cartoon head of Rick and Morty",   # 瑞克和莫蒂的卡通头像
    "an armchair in the shape of an avocado", # 一个牛油果形状的单人沙发
    "Fireworks are being set off at the Bird's Nest Stadium", # 鸟巢体育场正在燃放烟花
    "A Van Gogh-style shoulder bag", # 一个具有梵高风格的单肩包
]
# 输出
for prompt in prompts:
    print(prompt)
    text2image(prompt)

The cartoon head of Rick and Morty

an armchair in the shape of an avocado

Fireworks are being set off at the Bird's Nest Stadium

A Van Gogh-style shoulder bag

小结

项目仅是对dalle的简单尝试，更换为meta版本可以得到更高质量的结果，下面是meta版本生成的结果，更为贴近原始文本的语义，并且清晰度更高。

经过简单步骤，就可以脑洞大开的得制作一系列图片~

希望大家玩得开心~

作者:Armor

我在AI Studio上获得黄金等级，点亮6个徽章，来互关呀~ https://aistudio.baidu.com/aistudio/personalcenter/thirdview/392748

声明

此项目为搬运
原项目链接