c语言做网站账号登录系统,门户网站建设工作会议,服装外贸流程,wordpress首页自定义广告位又是好久没更新博客#xff0c;最近琐事缠身#xff0c;写文档写到吐。没时间学习新的知识#xff0c;刚空闲下来立刻就学习之前忘得差不多得Pytorch。Pytorch和tensorflow差不多#xff0c;具体得就不多啰嗦了#xff0c;觉得还有疑问的童鞋可以自行去官网学习#xff0…又是好久没更新博客最近琐事缠身写文档写到吐。没时间学习新的知识刚空闲下来立刻就学习之前忘得差不多得Pytorch。Pytorch和tensorflow差不多具体得就不多啰嗦了觉得还有疑问的童鞋可以自行去官网学习官网网址包括
1.https://pytorch.org/tutorials/beginner/deep_learning_60min_blitz.html
2.https://pytorch-cn.readthedocs.io/zh/latest/package_references/torch-nn/#class-torchnndataparallelmodule-device_idsnone-output_devicenone-dim0source 个人觉得值得一说的是关于数据并行这部分
根据官网的教程代码总结如下所示
# -*- coding: utf-8 -*-Created on Tue Apr 16 14:32:58 2019author: kofzh
import torch
import torch.nn as nn
from torch.utils.data import Dataset, DataLoader# Parameters and DataLoaders
input_size 5
output_size 2batch_size 30
data_size 100device torch.device(cuda:0 if torch.cuda.is_available() else cpu)class RandomDataset(Dataset):def __init__(self, size, length):self.len lengthself.data torch.randn(length, size)def __getitem__(self, index):return self.data[index]def __len__(self):return self.lenrand_loader DataLoader(datasetRandomDataset(input_size, data_size),batch_sizebatch_size, shuffleTrue)class Model(nn.Module):# Our modeldef __init__(self, input_size, output_size):super(Model, self).__init__()self.fc nn.Linear(input_size, output_size)def forward(self, input):output self.fc(input)print(\tIn Model: input size, input.size(),output size, output.size())return outputmodel Model(input_size, output_size)
if torch.cuda.device_count() 1:print(Lets use, torch.cuda.device_count(), GPUs!)# dim 0 [30, xxx] - [10, ...], [10, ...], [10, ...] on 3 GPUsmodel nn.DataParallel(model)model.to(device)
for data in rand_loader:input data.to(device)output model(input)print(Outside: input size, input.size(),output_size, output.size())
对于上述代码分别是 batchsize 30在CPU和4路GPU环境下运行得出的结果
CPU with batchsize 30结果
In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size torch.Size([30, 5]) output size torch.Size([30, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size torch.Size([10, 5]) output size torch.Size([10, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])
4路 GPU with batchsize 30 结果
Lets use 4 GPUs!In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])In Model: input size torch.Size([6, 5]) output size torch.Size([6, 2])In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size In Model: input sizetorch.Size([8, 5]) In Model: input size torch.Size([8, 5])output size In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])torch.Size([6, 5]) output size torch.Size([6, 2])output size torch.Size([8, 2])torch.Size([8, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])
torch.Size([8, 5]) output size torch.Size([8, 2])In Model: input size torch.Size([8, 5]) output size torch.Size([8, 2])In Model: input size torch.Size([6, 5]) output size torch.Size([6, 2])
Outside: input size torch.Size([30, 5]) output_size torch.Size([30, 2])In Model: input size In Model: input sizetorch.Size([3, 5]) In Model: input size torch.Size([3, 5]) output size torch.Size([3, 2])torch.Size([3, 5]) output size torch.Size([3, 2])output size torch.Size([3, 2])In Model: input size torch.Size([1, 5]) output size torch.Size([1, 2])
Outside: input size torch.Size([10, 5]) output_size torch.Size([10, 2])
对于上述代码分别是 batchsize 25在CPU和4路GPU环境下运行得出的结果
CPU with batchsize 25结果 In Model: input size torch.Size([25, 5]) output size torch.Size([25, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size torch.Size([25, 5]) output size torch.Size([25, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size torch.Size([25, 5]) output size torch.Size([25, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size torch.Size([25, 5]) output size torch.Size([25, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])
4路 GPU with batchsize 25 结果
Lets use 4 GPUs!In Model: input size In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])torch.Size([4, 5]) output size torch.Size([4, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])torch.Size([4, 5]) output size torch.Size([4, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])
torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
Outside: input size torch.Size([25, 5]) output_size torch.Size([25, 2])In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size In Model: input size torch.Size([7, 5]) output size torch.Size([7, 2])torch.Size([7, 5]) output size torch.Size([7, 2])In Model: input size torch.Size([4, 5]) output size torch.Size([4, 2])
个人总结batchsize在实际训练时对于每个GPU的分配基本上采用的是平均分配数据的原则每个GPU分配的数据相加 batchsize。
备注
1.官网上的CPU结果图是错误的可能是图截错了在此对于某些博客直接抄袭官网的结果图表示鄙视。
2.本文针对单服务器多GPU的情况请不要混淆。
如有错误敬请指正
