苏州建设招聘信息网站,网站信息系统,做旅游网站的工作流程图,大同工程造价信息网#xff08;beta#xff09;计算机视觉的量化迁移学习教程 原文#xff1a;pytorch.org/tutorials/intermediate/quantized_transfer_learning_tutorial.html 译者#xff1a;飞龙 协议#xff1a;CC BY-NC-SA 4.0 提示
为了充分利用本教程#xff0c;我们建议使用这个C…beta计算机视觉的量化迁移学习教程 原文pytorch.org/tutorials/intermediate/quantized_transfer_learning_tutorial.html 译者飞龙 协议CC BY-NC-SA 4.0 提示
为了充分利用本教程我们建议使用这个Colab 版本。这将允许您尝试下面提供的信息。
作者Zafar Takhirov
审阅者Raghuraman Krishnamoorthi
编辑Jessica Lin
本教程是基于原始的PyTorch 迁移学习教程构建的由Sasank Chilamkurthy编写。
迁移学习是指利用预训练模型应用于不同数据集的技术。迁移学习的主要使用方式有两种 将 ConvNet 作为固定特征提取器在这里您会“冻结”网络中除最后几层通常是完全连接的层也称为“头部”之外的所有参数的权重。这些最后的层将被新的层替换并用随机权重初始化只有这些层会被训练。 微调 ConvNet不是随机初始化而是使用预训练网络初始化模型然后训练过程与通常情况下不同数据集的训练相同。通常还会替换网络中的头部或其中的一部分以适应不同数量的输出。在这种方法中通常将学习率设置为较小的值。这是因为网络已经训练过只需要对其进行“微调”以适应新数据集。
您也可以结合上述两种方法首先可以冻结特征提取器并训练头部。之后您可以解冻特征提取器或其中的一部分将学习率设置为较小的值并继续训练。
在本部分中您将使用第一种方法——使用量化模型提取特征。
第 0 部分。先决条件
在深入研究迁移学习之前让我们回顾一下“先决条件”如安装和数据加载/可视化。
# Imports
import copy
import matplotlib.pyplot as plt
import numpy as np
import os
import timeplt.ion() 安装夜间版本
由于您将使用 PyTorch 的 beta 部分建议安装最新版本的torch和torchvision。您可以在本地安装的最新说明这里。例如要安装不带 GPU 支持的版本
pip install numpy
pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cpu/torch_nightly.html
# For CUDA support use https://download.pytorch.org/whl/nightly/cu101/torch_nightly.html 加载数据
注意
本节与原始迁移学习教程相同。
我们将使用torchvision和torch.utils.data包来加载数据。
今天您要解决的问题是从图像中对蚂蚁和蜜蜂进行分类。数据集包含大约 120 张蚂蚁和蜜蜂的训练图像。每个类别有 75 张验证图像。这被认为是一个非常小的数据集来进行泛化。但是由于我们使用迁移学习我们应该能够进行合理的泛化。
此数据集是 imagenet 的一个非常小的子集。
注意
从这里下载数据并将其解压缩到data目录中。
import torch
from torchvision import transforms, datasets# Data augmentation and normalization for training
# Just normalization for validation
data_transforms {train: transforms.Compose([transforms.Resize(224),transforms.RandomCrop(224),transforms.RandomHorizontalFlip(),transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]),val: transforms.Compose([transforms.Resize(224),transforms.CenterCrop(224),transforms.ToTensor(),transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])]),
}data_dir data/hymenoptera_data
image_datasets {x: datasets.ImageFolder(os.path.join(data_dir, x),data_transforms[x])for x in [train, val]}
dataloaders {x: torch.utils.data.DataLoader(image_datasets[x], batch_size16,shuffleTrue, num_workers8)for x in [train, val]}
dataset_sizes {x: len(image_datasets[x]) for x in [train, val]}
class_names image_datasets[train].classesdevice torch.device(cuda:0 if torch.cuda.is_available() else cpu) 可视化几张图片
让我们可视化一些训练图像以便了解数据增强。
import torchvisiondef imshow(inp, titleNone, axNone, figsize(5, 5)):Imshow for Tensor.inp inp.numpy().transpose((1, 2, 0))mean np.array([0.485, 0.456, 0.406])std np.array([0.229, 0.224, 0.225])inp std * inp meaninp np.clip(inp, 0, 1)if ax is None:fig, ax plt.subplots(1, figsizefigsize)ax.imshow(inp)ax.set_xticks([])ax.set_yticks([])if title is not None:ax.set_title(title)# Get a batch of training data
inputs, classes next(iter(dataloaders[train]))# Make a grid from batch
out torchvision.utils.make_grid(inputs, nrow4)fig, ax plt.subplots(1, figsize(10, 10))
imshow(out, title[class_names[x] for x in classes], axax) 用于模型训练的支持函数
以下是用于模型训练的通用函数。此函数还 调整学习率 保存最佳模型
def train_model(model, criterion, optimizer, scheduler, num_epochs25, devicecpu):Support function for model training.Args:model: Model to be trainedcriterion: Optimization criterion (loss)optimizer: Optimizer to use for trainingscheduler: Instance of torch.optim.lr_schedulernum_epochs: Number of epochsdevice: Device to run the training on. Must be cpu or cudasince time.time()best_model_wts copy.deepcopy(model.state_dict())best_acc 0.0for epoch in range(num_epochs):print(Epoch {}/{}.format(epoch, num_epochs - 1))print(- * 10)# Each epoch has a training and validation phasefor phase in [train, val]:if phase train:model.train() # Set model to training modeelse:model.eval() # Set model to evaluate moderunning_loss 0.0running_corrects 0# Iterate over data.for inputs, labels in dataloaders[phase]:inputs inputs.to(device)labels labels.to(device)# zero the parameter gradientsoptimizer.zero_grad()# forward# track history if only in trainwith torch.set_grad_enabled(phase train):outputs model(inputs)_, preds torch.max(outputs, 1)loss criterion(outputs, labels)# backward optimize only if in training phaseif phase train:loss.backward()optimizer.step()# statisticsrunning_loss loss.item() * inputs.size(0)running_corrects torch.sum(preds labels.data)if phase train:scheduler.step()epoch_loss running_loss / dataset_sizes[phase]epoch_acc running_corrects.double() / dataset_sizes[phase]print({} Loss: {:.4f} Acc: {:.4f}.format(phase, epoch_loss, epoch_acc))# deep copy the modelif phase val and epoch_acc best_acc:best_acc epoch_accbest_model_wts copy.deepcopy(model.state_dict())print()time_elapsed time.time() - sinceprint(Training complete in {:.0f}m {:.0f}s.format(time_elapsed // 60, time_elapsed % 60))print(Best val Acc: {:4f}.format(best_acc))# load best model weightsmodel.load_state_dict(best_model_wts)return model 用于可视化模型预测的支持函数
用于显示几张图片预测的通用函数
def visualize_model(model, rows3, cols3):was_training model.trainingmodel.eval()current_row current_col 0fig, ax plt.subplots(rows, cols, figsize(cols*2, rows*2))with torch.no_grad():for idx, (imgs, lbls) in enumerate(dataloaders[val]):imgs imgs.cpu()lbls lbls.cpu()outputs model(imgs)_, preds torch.max(outputs, 1)for jdx in range(imgs.size()[0]):imshow(imgs.data[jdx], axax[current_row, current_col])ax[current_row, current_col].axis(off)ax[current_row, current_col].set_title(predicted: {}.format(class_names[preds[jdx]]))current_col 1if current_col cols:current_row 1current_col 0if current_row rows:model.train(modewas_training)returnmodel.train(modewas_training) 第 1 部分。基于量化特征提取器训练自定义分类器
在本节中您将使用一个“冻结”的可量化特征提取器并在其顶部训练一个自定义分类器头。与浮点模型不同您不需要为可量化模型设置 requires_gradFalse因为它没有可训练的参数。请参考文档以获取更多详细信息。
加载预训练模型在本练习中您将使用ResNet-18。
import torchvision.models.quantization as models# You will need the number of filters in the fc for future use.
# Here the size of each output sample is set to 2.
# Alternatively, it can be generalized to nn.Linear(num_ftrs, len(class_names)).
model_fe models.resnet18(pretrainedTrue, progressTrue, quantizeTrue)
num_ftrs model_fe.fc.in_features 此时您需要修改预训练模型。该模型在开头和结尾有量化/去量化块。但是因为您只会使用特征提取器所以去量化层必须移动到线性层头部的右侧。最简单的方法是将模型包装在nn.Sequential模块中。
第一步是在 ResNet 模型中隔离特征提取器。尽管在这个例子中您被要求使用除fc之外的所有层作为特征提取器但实际上您可以取需要的部分。这在您想要替换一些卷积层时会很有用。
注意
当将特征提取器与量化模型的其余部分分离时您必须手动将量化器/去量化器放置在您想要保持量化的部分的开头和结尾。
下面的函数创建了一个带有自定义头的模型。
from torch import nndef create_combined_model(model_fe):# Step 1\. Isolate the feature extractor.model_fe_features nn.Sequential(model_fe.quant, # Quantize the inputmodel_fe.conv1,model_fe.bn1,model_fe.relu,model_fe.maxpool,model_fe.layer1,model_fe.layer2,model_fe.layer3,model_fe.layer4,model_fe.avgpool,model_fe.dequant, # Dequantize the output)# Step 2\. Create a new headnew_head nn.Sequential(nn.Dropout(p0.5),nn.Linear(num_ftrs, 2),)# Step 3\. Combine, and dont forget the quant stubs.new_model nn.Sequential(model_fe_features,nn.Flatten(1),new_head,)return new_model 警告
目前量化模型只能在 CPU 上运行。但是可以将模型的非量化部分发送到 GPU 上。
import torch.optim as optim
new_model create_combined_model(model_fe)
new_model new_model.to(cpu)criterion nn.CrossEntropyLoss()# Note that we are only training the head.
optimizer_ft optim.SGD(new_model.parameters(), lr0.01, momentum0.9)# Decay LR by a factor of 0.1 every 7 epochs
exp_lr_scheduler optim.lr_scheduler.StepLR(optimizer_ft, step_size7, gamma0.1) 训练和评估
这一步在 CPU 上大约需要 15-25 分钟。由于量化模型只能在 CPU 上运行因此无法在 GPU 上运行训练。
new_model train_model(new_model, criterion, optimizer_ft, exp_lr_scheduler,num_epochs25, devicecpu)visualize_model(new_model)
plt.tight_layout() 第 2 部分。微调可量化模型
在这部分中我们微调用于迁移学习的特征提取器并对特征提取器进行量化。请注意在第 1 部分和第 2 部分中特征提取器都被量化。不同之处在于在第 1 部分中我们使用了一个预训练的量化模型。而在这部分中我们在感兴趣的数据集上微调后创建了一个量化的特征提取器因此这是一种在迁移学习中获得更好准确性并具有量化优势的方法。请注意在我们的具体示例中训练集非常小120 张图像因此整个模型微调的好处并不明显。然而这里展示的过程将提高在具有更大数据集的迁移学习中的准确性。
预训练的特征提取器必须是可量化的。为确保它是可量化的请执行以下步骤 使用torch.quantization.fuse_modules融合(Conv, BN, ReLU)、(Conv, BN)和(Conv, ReLU)。将特征提取器与自定义头连接。这需要对特征提取器的输出进行去量化。在特征提取器的适当位置插入伪量化模块以在训练过程中模拟量化。 对于第1步我们使用torchvision/models/quantization中的模型这些模型具有成员方法fuse_model。此函数将所有conv、bn和relu模块融合在一起。对于自定义模型这将需要手动调用torch.quantization.fuse_modules API并提供要手动融合的模块列表。
第2步由前一节中使用的create_combined_model函数执行。
第3步通过使用torch.quantization.prepare_qat来实现该函数插入了伪量化模块。
作为第4步您可以开始“微调”模型然后将其转换为完全量化的版本第 5 步。
要将微调后的模型转换为量化模型您可以调用torch.quantization.convert函数在我们的情况下只有特征提取器被量化。
注意
由于随机初始化您的结果可能与本教程中显示的结果不同。
# notice quantizeFalse
model models.resnet18(pretrainedTrue, progressTrue, quantizeFalse)
num_ftrs model.fc.in_features# Step 1
model.train()
model.fuse_model()
# Step 2
model_ft create_combined_model(model)
model_ft[0].qconfig torch.quantization.default_qat_qconfig # Use default QAT configuration
# Step 3
model_ft torch.quantization.prepare_qat(model_ft, inplaceTrue) 微调模型
在当前教程中整个模型都被微调。一般来说这会导致更高的准确性。然而由于这里使用的训练集很小我们最终会过拟合训练集。
步骤 4. 微调模型
for param in model_ft.parameters():param.requires_grad Truemodel_ft.to(device) # We can fine-tune on GPU if availablecriterion nn.CrossEntropyLoss()# Note that we are training everything, so the learning rate is lower
# Notice the smaller learning rate
optimizer_ft optim.SGD(model_ft.parameters(), lr1e-3, momentum0.9, weight_decay0.1)# Decay LR by a factor of 0.3 every several epochs
exp_lr_scheduler optim.lr_scheduler.StepLR(optimizer_ft, step_size5, gamma0.3)model_ft_tuned train_model(model_ft, criterion, optimizer_ft, exp_lr_scheduler,num_epochs25, devicedevice) 步骤 5. 转换为量化模型
from torch.quantization import convert
model_ft_tuned.cpu()model_quantized_and_trained convert(model_ft_tuned, inplaceFalse) 让我们看看量化模型在几张图片上的表现
visualize_model(model_quantized_and_trained)plt.ioff()
plt.tight_layout()
plt.show() beta在 PyTorch 中使用急切模式的静态量化 原文pytorch.org/tutorials/advanced/static_quantization_tutorial.html 译者飞龙 协议CC BY-NC-SA 4.0 作者Raghuraman Krishnamoorthi 编辑Seth WeidmanJerry Zhang
本教程展示了如何进行后训练静态量化以及展示了两种更高级的技术 - 按通道量化和量化感知训练 - 以进一步提高模型的准确性。请注意目前量化仅支持 CPU因此在本教程中我们将不使用 GPU/CUDA。通过本教程您将看到 PyTorch 中的量化如何导致模型尺寸显著减小同时提高速度。此外您将看到如何轻松应用一些高级量化技术使您的量化模型比以往更少地准确性下降。警告我们从其他 PyTorch 存储库中使用了大量样板代码例如定义MobileNetV2模型架构定义数据加载器等。当然我们鼓励您阅读它但如果您想了解量化功能请随时跳到“4. 后训练静态量化”部分。我们将从进行必要的导入开始
import os
import sys
import time
import numpy as npimport torch
import torch.nn as nn
from torch.utils.data import DataLoaderimport torchvision
from torchvision import datasets
import torchvision.transforms as transforms# Set up warnings
import warnings
warnings.filterwarnings(actionignore,categoryDeprecationWarning,moduler.*
)
warnings.filterwarnings(actiondefault,modulertorch.ao.quantization
)# Specify random seed for repeatable results
torch.manual_seed(191009) 1. 模型架构
我们首先定义 MobileNetV2 模型架构进行了几个显著的修改以实现量化 用nn.quantized.FloatFunctional替换加法 在网络的开头和结尾插入QuantStub和DeQuantStub。 用 ReLU 替换 ReLU6
注意此代码取自这里。
from torch.ao.quantization import QuantStub, DeQuantStubdef _make_divisible(v, divisor, min_valueNone):This function is taken from the original tf repo.It ensures that all layers have a channel number that is divisible by 8It can be seen here:https://github.com/tensorflow/models/blob/master/research/slim/nets/mobilenet/mobilenet.py:param v::param divisor::param min_value::return:if min_value is None:min_value divisornew_v max(min_value, int(v divisor / 2) // divisor * divisor)# Make sure that round down does not go down by more than 10%.if new_v 0.9 * v:new_v divisorreturn new_vclass ConvBNReLU(nn.Sequential):def __init__(self, in_planes, out_planes, kernel_size3, stride1, groups1):padding (kernel_size - 1) // 2super(ConvBNReLU, self).__init__(nn.Conv2d(in_planes, out_planes, kernel_size, stride, padding, groupsgroups, biasFalse),nn.BatchNorm2d(out_planes, momentum0.1),# Replace with ReLUnn.ReLU(inplaceFalse))class InvertedResidual(nn.Module):def __init__(self, inp, oup, stride, expand_ratio):super(InvertedResidual, self).__init__()self.stride strideassert stride in [1, 2]hidden_dim int(round(inp * expand_ratio))self.use_res_connect self.stride 1 and inp ouplayers []if expand_ratio ! 1:# pwlayers.append(ConvBNReLU(inp, hidden_dim, kernel_size1))layers.extend([# dwConvBNReLU(hidden_dim, hidden_dim, stridestride, groupshidden_dim),# pw-linearnn.Conv2d(hidden_dim, oup, 1, 1, 0, biasFalse),nn.BatchNorm2d(oup, momentum0.1),])self.conv nn.Sequential(*layers)# Replace torch.add with floatfunctionalself.skip_add nn.quantized.FloatFunctional()def forward(self, x):if self.use_res_connect:return self.skip_add.add(x, self.conv(x))else:return self.conv(x)class MobileNetV2(nn.Module):def __init__(self, num_classes1000, width_mult1.0, inverted_residual_settingNone, round_nearest8):MobileNet V2 main classArgs:num_classes (int): Number of classeswidth_mult (float): Width multiplier - adjusts number of channels in each layer by this amountinverted_residual_setting: Network structureround_nearest (int): Round the number of channels in each layer to be a multiple of this numberSet to 1 to turn off roundingsuper(MobileNetV2, self).__init__()block InvertedResidualinput_channel 32last_channel 1280if inverted_residual_setting is None:inverted_residual_setting [# t, c, n, s[1, 16, 1, 1],[6, 24, 2, 2],[6, 32, 3, 2],[6, 64, 4, 2],[6, 96, 3, 1],[6, 160, 3, 2],[6, 320, 1, 1],]# only check the first element, assuming user knows t,c,n,s are requiredif len(inverted_residual_setting) 0 or len(inverted_residual_setting[0]) ! 4:raise ValueError(inverted_residual_setting should be non-empty or a 4-element list, got {}.format(inverted_residual_setting))# building first layerinput_channel _make_divisible(input_channel * width_mult, round_nearest)self.last_channel _make_divisible(last_channel * max(1.0, width_mult), round_nearest)features [ConvBNReLU(3, input_channel, stride2)]# building inverted residual blocksfor t, c, n, s in inverted_residual_setting:output_channel _make_divisible(c * width_mult, round_nearest)for i in range(n):stride s if i 0 else 1features.append(block(input_channel, output_channel, stride, expand_ratiot))input_channel output_channel# building last several layersfeatures.append(ConvBNReLU(input_channel, self.last_channel, kernel_size1))# make it nn.Sequentialself.features nn.Sequential(*features)self.quant QuantStub()self.dequant DeQuantStub()# building classifierself.classifier nn.Sequential(nn.Dropout(0.2),nn.Linear(self.last_channel, num_classes),)# weight initializationfor m in self.modules():if isinstance(m, nn.Conv2d):nn.init.kaiming_normal_(m.weight, modefan_out)if m.bias is not None:nn.init.zeros_(m.bias)elif isinstance(m, nn.BatchNorm2d):nn.init.ones_(m.weight)nn.init.zeros_(m.bias)elif isinstance(m, nn.Linear):nn.init.normal_(m.weight, 0, 0.01)nn.init.zeros_(m.bias)def forward(self, x):x self.quant(x)x self.features(x)x x.mean([2, 3])x self.classifier(x)x self.dequant(x)return x# Fuse ConvBN and ConvBNRelu modules prior to quantization# This operation does not change the numericsdef fuse_model(self, is_qatFalse):fuse_modules torch.ao.quantization.fuse_modules_qat if is_qat else torch.ao.quantization.fuse_modulesfor m in self.modules():if type(m) ConvBNReLU:fuse_modules(m, [0, 1, 2], inplaceTrue)if type(m) InvertedResidual:for idx in range(len(m.conv)):if type(m.conv[idx]) nn.Conv2d:fuse_modules(m.conv, [str(idx), str(idx 1)], inplaceTrue) 2. 辅助函数
接下来我们定义了几个辅助函数来帮助模型评估。这些大部分来自这里。
class AverageMeter(object):Computes and stores the average and current valuedef __init__(self, name, fmt:f):self.name nameself.fmt fmtself.reset()def reset(self):self.val 0self.avg 0self.sum 0self.count 0def update(self, val, n1):self.val valself.sum val * nself.count nself.avg self.sum / self.countdef __str__(self):fmtstr {name} {val self.fmt } ({avg self.fmt })return fmtstr.format(**self.__dict__)def accuracy(output, target, topk(1,)):Computes the accuracy over the k top predictions for the specified values of kwith torch.no_grad():maxk max(topk)batch_size target.size(0)_, pred output.topk(maxk, 1, True, True)pred pred.t()correct pred.eq(target.view(1, -1).expand_as(pred))res []for k in topk:correct_k correct[:k].reshape(-1).float().sum(0, keepdimTrue)res.append(correct_k.mul_(100.0 / batch_size))return resdef evaluate(model, criterion, data_loader, neval_batches):model.eval()top1 AverageMeter(Acc1, :6.2f)top5 AverageMeter(Acc5, :6.2f)cnt 0with torch.no_grad():for image, target in data_loader:output model(image)loss criterion(output, target)cnt 1acc1, acc5 accuracy(output, target, topk(1, 5))print(., end )top1.update(acc1[0], image.size(0))top5.update(acc5[0], image.size(0))if cnt neval_batches:return top1, top5return top1, top5def load_model(model_file):model MobileNetV2()state_dict torch.load(model_file)model.load_state_dict(state_dict)model.to(cpu)return modeldef print_size_of_model(model):torch.save(model.state_dict(), temp.p)print(Size (MB):, os.path.getsize(temp.p)/1e6)os.remove(temp.p) 3. 定义数据集和数据加载器
作为我们最后的主要设置步骤我们为训练集和测试集定义数据加载器。
ImageNet 数据
要使用整个 ImageNet 数据集运行本教程中的代码请首先按照ImageNet 数据中的说明下载 imagenet。将下载的文件解压缩到“data_path”文件夹中。
数据下载完成后我们展示下面的函数定义数据加载器我们将使用这些数据读取。这些函数大部分来自这里。
def prepare_data_loaders(data_path):normalize transforms.Normalize(mean[0.485, 0.456, 0.406],std[0.229, 0.224, 0.225])dataset torchvision.datasets.ImageNet(data_path, splittrain, transformtransforms.Compose([transforms.RandomResizedCrop(224),transforms.RandomHorizontalFlip(),transforms.ToTensor(),normalize,]))dataset_test torchvision.datasets.ImageNet(data_path, splitval, transformtransforms.Compose([transforms.Resize(256),transforms.CenterCrop(224),transforms.ToTensor(),normalize,]))train_sampler torch.utils.data.RandomSampler(dataset)test_sampler torch.utils.data.SequentialSampler(dataset_test)data_loader torch.utils.data.DataLoader(dataset, batch_sizetrain_batch_size,samplertrain_sampler)data_loader_test torch.utils.data.DataLoader(dataset_test, batch_sizeeval_batch_size,samplertest_sampler)return data_loader, data_loader_test 接下来我们将加载预训练的 MobileNetV2 模型。我们提供了下载模型的 URL here。
data_path ~/.data/imagenet
saved_model_dir data/
float_model_file mobilenet_pretrained_float.pth
scripted_float_model_file mobilenet_quantization_scripted.pth
scripted_quantized_model_file mobilenet_quantization_scripted_quantized.pthtrain_batch_size 30
eval_batch_size 50data_loader, data_loader_test prepare_data_loaders(data_path)
criterion nn.CrossEntropyLoss()
float_model load_model(saved_model_dir float_model_file).to(cpu)# Next, well fuse modules; this can both make the model faster by saving on memory access
# while also improving numerical accuracy. While this can be used with any model, this is
# especially common with quantized models.print(\n Inverted Residual Block: Before fusion \n\n, float_model.features[1].conv)
float_model.eval()# Fuses modules
float_model.fuse_model()# Note fusion of ConvBNRelu and ConvRelu
print(\n Inverted Residual Block: After fusion\n\n,float_model.features[1].conv) 最后为了获得“基线”准确性让我们看看我们未量化模型与融合模块的准确性。
num_eval_batches 1000print(Size of baseline model)
print_size_of_model(float_model)top1, top5 evaluate(float_model, criterion, data_loader_test, neval_batchesnum_eval_batches)
print(Evaluation accuracy on %d images, %2.2f%(num_eval_batches * eval_batch_size, top1.avg))
torch.jit.save(torch.jit.script(float_model), saved_model_dir scripted_float_model_file) 在整个模型上我们在包含 50,000 张图像的评估数据集上获得了 71.9%的准确率。
这将是我们用来比较的基线。接下来让我们尝试不同的量化方法
4. 后训练静态量化
静态后训练量化不仅涉及将权重从浮点转换为整数如动态量化那样还包括首先通过网络传递数据批次并计算不同激活的结果分布的额外步骤具体来说这是通过在不同点插入观察器模块记录这些数据来完成的。然后使用这些分布来确定如何在推断时量化不同的激活一个简单的技术是将整个激活范围分成 256 个级别但我们也支持更复杂的方法。重要的是这个额外的步骤允许我们在操作之间传递量化值而不是在每个操作之间将这些值转换为浮点数 - 然后再转换为整数从而实现显著的加速。
num_calibration_batches 32myModel load_model(saved_model_dir float_model_file).to(cpu)
myModel.eval()# Fuse Conv, bn and relu
myModel.fuse_model()# Specify quantization configuration
# Start with simple min/max range estimation and per-tensor quantization of weights
myModel.qconfig torch.ao.quantization.default_qconfig
print(myModel.qconfig)
torch.ao.quantization.prepare(myModel, inplaceTrue)# Calibrate first
print(Post Training Quantization Prepare: Inserting Observers)
print(\n Inverted Residual Block:After observer insertion \n\n, myModel.features[1].conv)# Calibrate with the training set
evaluate(myModel, criterion, data_loader, neval_batchesnum_calibration_batches)
print(Post Training Quantization: Calibration done)# Convert to quantized model
torch.ao.quantization.convert(myModel, inplaceTrue)
# You may see a user warning about needing to calibrate the model. This warning can be safely ignored.
# This warning occurs because not all modules are run in each model runs, so some
# modules may not be calibrated.
print(Post Training Quantization: Convert done)
print(\n Inverted Residual Block: After fusion and quantization, note fused modules: \n\n,myModel.features[1].conv)print(Size of model after quantization)
print_size_of_model(myModel)top1, top5 evaluate(myModel, criterion, data_loader_test, neval_batchesnum_eval_batches)
print(Evaluation accuracy on %d images, %2.2f%(num_eval_batches * eval_batch_size, top1.avg)) 对于这个量化模型在评估数据集上看到准确率为 56.7%。这是因为我们使用简单的最小/最大观察器来确定量化参数。尽管如此我们将模型的大小减小到了将近 3.6 MB几乎减少了 4 倍。
此外我们可以通过使用不同的量化配置显着提高准确性。我们使用为 x86 架构量化推荐的配置重复相同的练习。此配置执行以下操作 按通道对权重进行量化 使用直方图观察器收集激活的直方图然后以最佳方式选择量化参数。
per_channel_quantized_model load_model(saved_model_dir float_model_file)
per_channel_quantized_model.eval()
per_channel_quantized_model.fuse_model()
# The old fbgemm is still available but x86 is the recommended default.
per_channel_quantized_model.qconfig torch.ao.quantization.get_default_qconfig(x86)
print(per_channel_quantized_model.qconfig)torch.ao.quantization.prepare(per_channel_quantized_model, inplaceTrue)
evaluate(per_channel_quantized_model,criterion, data_loader, num_calibration_batches)
torch.ao.quantization.convert(per_channel_quantized_model, inplaceTrue)
top1, top5 evaluate(per_channel_quantized_model, criterion, data_loader_test, neval_batchesnum_eval_batches)
print(Evaluation accuracy on %d images, %2.2f%(num_eval_batches * eval_batch_size, top1.avg))
torch.jit.save(torch.jit.script(per_channel_quantized_model), saved_model_dir scripted_quantized_model_file) 仅改变这种量化配置方法就使准确率提高到 67.3%以上但是这仍然比上面实现的 71.9%的基准差 4%。所以让我们尝试量化感知训练。
5. 量化感知训练
量化感知训练QAT通常是产生最高准确率的量化方法。使用 QAT 时在训练的前向和后向传递中所有权重和激活都被“伪量化”即浮点值被四舍五入以模拟 int8 值但所有计算仍然使用浮点数。因此在训练期间进行的所有权重调整都是在“意识到”模型最终将被量化的情况下进行的因此在量化之后这种方法通常会产生比动态量化或静态后训练量化更高的准确率。
实际执行 QAT 的整体工作流程与以前非常相似 我们可以使用与以前相同的模型对于量化感知训练不需要额外的准备工作。 我们需要使用一个qconfig指定在权重和激活之后要插入什么样的伪量化而不是指定观察器
我们首先定义一个训练函数
def train_one_epoch(model, criterion, optimizer, data_loader, device, ntrain_batches):model.train()top1 AverageMeter(Acc1, :6.2f)top5 AverageMeter(Acc5, :6.2f)avgloss AverageMeter(Loss, 1.5f)cnt 0for image, target in data_loader:start_time time.time()print(., end )cnt 1image, target image.to(device), target.to(device)output model(image)loss criterion(output, target)optimizer.zero_grad()loss.backward()optimizer.step()acc1, acc5 accuracy(output, target, topk(1, 5))top1.update(acc1[0], image.size(0))top5.update(acc5[0], image.size(0))avgloss.update(loss, image.size(0))if cnt ntrain_batches:print(Loss, avgloss.avg)print(Training: * Acc1 {top1.avg:.3f} Acc5 {top5.avg:.3f}.format(top1top1, top5top5))returnprint(Full imagenet train set: * Acc1 {top1.global_avg:.3f} Acc5 {top5.global_avg:.3f}.format(top1top1, top5top5))return 我们像以前一样融合模块
qat_model load_model(saved_model_dir float_model_file)
qat_model.fuse_model(is_qatTrue)optimizer torch.optim.SGD(qat_model.parameters(), lr 0.0001)
# The old fbgemm is still available but x86 is the recommended default.
qat_model.qconfig torch.ao.quantization.get_default_qat_qconfig(x86) 最后prepare_qat执行“伪量化”为量化感知训练准备模型
torch.ao.quantization.prepare_qat(qat_model, inplaceTrue)
print(Inverted Residual Block: After preparation for QAT, note fake-quantization modules \n,qat_model.features[1].conv) 训练一个准确率高的量化模型需要准确地模拟推断时的数值。因此对于量化感知训练我们通过修改训练循环来进行 在训练结束时切换批量归一化以使用运行时均值和方差以更好地匹配推断数值。 我们还会冻结量化器参数比例和零点并微调权重。
num_train_batches 20# QAT takes time and one needs to train over a few epochs.
# Train and check accuracy after each epoch
for nepoch in range(8):train_one_epoch(qat_model, criterion, optimizer, data_loader, torch.device(cpu), num_train_batches)if nepoch 3:# Freeze quantizer parametersqat_model.apply(torch.ao.quantization.disable_observer)if nepoch 2:# Freeze batch norm mean and variance estimatesqat_model.apply(torch.nn.intrinsic.qat.freeze_bn_stats)# Check the accuracy after each epochquantized_model torch.ao.quantization.convert(qat_model.eval(), inplaceFalse)quantized_model.eval()top1, top5 evaluate(quantized_model,criterion, data_loader_test, neval_batchesnum_eval_batches)print(Epoch %d :Evaluation accuracy on %d images, %2.2f%(nepoch, num_eval_batches * eval_batch_size, top1.avg)) 量化感知训练在整个 imagenet 数据集上的准确率超过 71.5%接近 71.9%的浮点准确率。
更多关于量化感知训练 QAT 是一种超集包含了更多的调试后量化技术。例如我们可以分析模型的准确性是否受到权重或激活量化的限制。 由于我们使用伪量化来模拟实际量化算术的数值因此我们还可以模拟量化模型在浮点数上的准确性。 我们也可以轻松地模拟后训练量化。
从量化中加速
最后让我们确认我们上面提到的一点我们的量化模型是否确实执行推断更快让我们测试一下
def run_benchmark(model_file, img_loader):elapsed 0model torch.jit.load(model_file)model.eval()num_batches 5# Run the scripted model on a few batches of imagesfor i, (images, target) in enumerate(img_loader):if i num_batches:start time.time()output model(images)end time.time()elapsed elapsed (end-start)else:breaknum_images images.size()[0] * num_batchesprint(Elapsed time: %3.0f ms % (elapsed/num_images*1000))return elapsedrun_benchmark(saved_model_dir scripted_float_model_file, data_loader_test)run_benchmark(saved_model_dir scripted_quantized_model_file, data_loader_test) 在 MacBook Pro 上本地运行常规模型的运行时间为 61 毫秒量化模型仅为 20 毫秒显示了与浮点模型相比我们通常看到的 2-4 倍加速。
结论
在本教程中我们展示了两种量化方法 - 后训练静态量化和量化感知训练 - 描述了它们在 PyTorch 中的使用方法。
感谢阅读我们始终欢迎任何反馈意见如果您有任何问题请在这里创建一个问题。
从第一原则理解 PyTorch Intel CPU 性能 原文pytorch.org/tutorials/intermediate/torchserve_with_ipex.html 译者飞龙 协议CC BY-NC-SA 4.0 一个关于使用Intel® PyTorch 扩展*优化的 TorchServe 推理框架的案例研究。
作者Min Jean Cho, Mark Saroufim
审阅者Ashok Emani, Jiong Gong
在 CPU 上获得强大的深度学习开箱即用性能可能有些棘手但如果您了解影响性能的主要问题如何衡量它们以及如何解决它们那么就会更容易。
简而言之
问题如何衡量解决方案瓶颈的 GEMM 执行单元 不平衡或串行旋转 前端受限 核心受限
通过核心固定将线程亲和性设置为物理核心以避免使用逻辑核心非均匀内存访问NUMA 本地与远程内存访问 UPI 利用率 内存访问延迟 线程迁移
通过核心固定将线程亲和性设置为特定插槽以避免跨插槽计算
GEMM通用矩阵乘法在融合乘加FMA或点积DP执行单元上运行这将成为瓶颈并在启用超线程时导致线程等待/旋转在同步屏障时出现延迟 - 因为使用逻辑核心会导致所有工作线程的并发性不足因为每个逻辑线程争夺相同的核心资源。相反如果我们每个物理核心使用 1 个线程我们就可以避免这种争夺。因此我们通常建议通过核心固定将 CPU 线程亲和性设置为物理核心从而避免逻辑核心。
多插槽系统具有非均匀内存访问NUMA这是一种共享内存架构描述了主存储模块相对于处理器的放置方式。但是如果一个进程不是 NUMA 感知的那么在运行时当线程通过Intel Ultra Path Interconnect (UPI)跨插槽迁移时会频繁访问慢远程内存。我们通过将 CPU 线程亲和性设置为特定插槽以通过核心固定解决这个问题。
牢记这些原则适当的 CPU 运行时配置可以显著提升开箱即用的性能。
在这篇博客中我们将带您了解CPU 性能调优指南中应该注意的重要运行时配置解释它们的工作原理如何对其进行分析以及如何将其集成到像TorchServe这样的模型服务框架中通过一个易于使用的启动脚本我们已经原生地集成了。
我们将从第一原则以及大量概要中直观地解释所有这些想法并向您展示我们如何应用我们的学习以改善 TorchServe 的开箱即用 CPU 性能。
必须通过在config.properties中设置cpu_launcher_enabletrue来显式启用该功能。
避免逻辑核心进行深度学习
通常避免逻辑核心进行深度学习工作负载会提高性能。为了理解这一点让我们回到 GEMM。
优化 GEMM 优化深度学习
深度学习训练或推断中的大部分时间都花在了 GEMM 的数百万次重复操作上这是完全连接层的核心。自从多层感知器MLP被证明是任何连续函数的通用逼近器以来完全连接层已经被使用了几十年。任何 MLP 都可以完全表示为 GEMM。甚至卷积也可以通过使用Toepliz 矩阵表示为 GEMM。
回到原来的话题大多数 GEMM 运算符受益于使用非超线程因为深度学习训练或推断中的大部分时间都花在了运行在超线程核心上的融合乘加FMA或点积DP执行单元上的数百万次重复操作上。启用超线程后OpenMP 线程将争夺相同的 GEMM 执行单元。 如果 2 个逻辑线程同时运行 GEMM它们将共享相同的核心资源导致前端绑定这样前端绑定带来的开销大于同时运行两个逻辑线程带来的收益。
因此我们通常建议避免在深度学习工作负载中使用逻辑核心以获得良好的性能。默认情况下启动脚本仅使用物理核心但是用户可以通过简单切换--use_logical_core启动脚本旋钮来轻松尝试逻辑核心与物理核心。
练习
我们将使用以下示例来提供 ResNet50 虚拟张量
import torch
import torchvision.models as models
import timemodel models.resnet50(pretrainedFalse)
model.eval()
data torch.rand(1, 3, 224, 224)# warm up
for _ in range(100):model(data)start time.time()
for _ in range(100):model(data)
end time.time()
print(Inference took {:.2f} ms in average.format((end-start)/100*1000)) 在博客中我们将使用Intel® VTune™ Profiler来进行分析和验证优化。我们将在一台配备两个 Intel® Xeon® Platinum 8180M CPU 的机器上运行所有练习。CPU 信息如图 2.1 所示。
环境变量OMP_NUM_THREADS用于设置并行区域的线程数。我们将比较OMP_NUM_THREADS2与1使用逻辑核心和2仅使用物理核心。
两个 OpenMP 线程尝试利用由超线程核心0, 56共享的相同 GEMM 执行单元
我们可以通过在 Linux 上运行htop命令来可视化这一点。 我们注意到旋转时间被标记并且不平衡或串行旋转占据了大部分时间 - 在总共 8.982 秒中的 4.980 秒。使用逻辑核心时的不平衡或串行旋转是由于工作线程的并发性不足因为每个逻辑线程争夺相同的核心资源。
执行摘要的 Top Hotspots 部分显示__kmp_fork_barrier占用了 4.589 秒的 CPU 时间 - 在 CPU 执行时间的 9.33%期间线程在这个屏障处旋转以进行线程同步。
每个 OpenMP 线程利用各自物理核心0,1中的 GEMM 执行单元 我们首先注意到通过避免逻辑核心执行时间从 32 秒降至 23 秒。虽然仍存在一些不可忽略的不平衡或串行旋转但我们注意到从 4.980 秒到 3.887 秒的相对改善。
通过不使用逻辑线程而是每个物理核心使用 1 个线程我们避免了逻辑线程争夺相同核心资源。Top Hotspots 部分还显示了__kmp_fork_barrier时间从 4.589 秒改善到 3.530 秒的相对改善。
本地内存访问始终比远程内存访问快。
通常我们建议将一个进程绑定到本地插槽以便该进程不会在不同插槽之间迁移。通常这样做的目的是利用本地内存上的高速缓存并避免远程内存访问后者可能慢大约 2 倍。 图 1. 双插槽配置
图 1. 显示了一个典型的双插槽配置。请注意每个插槽都有自己的本地内存。插槽通过 Intel Ultra Path Interconnect (UPI)连接到彼此这允许每个插槽访问另一个插槽的本地内存称为远程内存。本地内存访问始终比远程内存访问快。 图 2.1. CPU 信息
用户可以通过在他们的 Linux 机器上运行lscpu命令来获取他们的 CPU 信息。图 2.1. 显示了在一台装有两个 Intel® Xeon® Platinum 8180M CPU 的机器上执行lscpu的示例。请注意每个插槽有 28 个核心每个核心有 2 个线程即启用了超线程。换句话说除了 28 个物理核心外还有 28 个逻辑核心每个插槽总共有 56 个核心。而且有 2 个插槽总共有 112 个核心每个核心的线程数 x 每个插槽的核心数 x 插槽数。 图 2.2. CPU 信息
这 2 个插槽分别映射到 2 个 NUMA 节点NUMA 节点 0NUMA 节点 1。物理核心在逻辑核心之前进行索引。如图 2.2.所示第一个插槽上的前 28 个物理核心0-27和前 28 个逻辑核心56-83位于 NUMA 节点 0。第二个插槽上的第 28 个物理核心28-55和第二个插槽上的第 28 个逻辑核心84-111位于 NUMA 节点 1。同一插槽上的核心共享本地内存和最后一级缓存LLC比通过 Intel UPI 进行跨插槽通信要快得多。
现在我们了解了 NUMA、跨插槽UPI流量、多处理器系统中的本地与远程内存访问让我们对其进行分析和验证。
练习
我们将重用上面的 ResNet50 示例。
由于我们没有将线程固定到特定插槽的处理器核心上操作系统会定期将线程调度到位于不同插槽中的处理器核心上。 图 3. 非 NUMA 感知应用程序的 CPU 使用情况。启动了 1 个主工作线程然后在所有核心上启动了一个物理核心编号56的线程包括逻辑核心。
附注如果线程数未通过torch.set_num_threads设置那么默认线程数是启用超线程系统中的物理核心数。这可以通过torch.get_num_threads来验证。因此我们看到大约一半的核心忙于运行示例脚本。 图 4. 非均匀内存访问分析图
图 4. 比较了随时间变化的本地与远程内存访问。我们验证了远程内存的使用这可能导致性能不佳。
设置线程亲和性以减少远程内存访问和跨插槽UPI流量
将线程固定到同一插槽上的核心有助于保持内存访问的局部性。在这个例子中我们将固定到第一个 NUMA 节点上的物理核心0-27。通过启动脚本用户可以通过简单切换--node_id启动脚本旋钮来轻松尝试 NUMA 节点配置。
让我们现在来可视化 CPU 使用情况。 图 5. NUMA 感知应用程序的 CPU 使用情况
启动了 1 个主要工作线程然后在第一个 NUMA 节点上的所有物理核心上启动了线程。 图 6. 非均匀内存访问分析图
如图 6 所示现在几乎所有的内存访问都是本地访问。
通过核心固定实现多工作线程推理的高效 CPU 使用率
在运行多工作线程推理时核心在工作线程之间重叠或共享导致 CPU 使用效率低下。为了解决这个问题启动脚本将可用核心数均等地分配给每个工作线程使每个工作线程在运行时固定到分配的核心。
使用 TorchServe 进行练习
在这个练习中让我们将我们迄今讨论过的 CPU 性能调优原则和建议应用到TorchServe apache-bench 基准测试。
我们将使用 ResNet50 进行 4 个工作线程并发数为 100请求为 10,000。所有其他参数例如batch_size输入等与默认参数相同。
我们将比较以下三种配置 默认的 TorchServe 设置没有核心固定 torch.set_num_threads 物理核心数 / 工作线程数没有核心固定 通过启动脚本进行核心固定要求 Torchserve0.6.1
经过这个练习我们将验证我们更喜欢避免逻辑核心而是通过核心固定实现本地内存访问使用真实的 TorchServe 用例。
1. 默认的 TorchServe 设置没有核心固定
base_handler没有明确设置torch.set_num_threads。因此默认线程数是物理 CPU 核心数如这里所述。用户可以通过在 base_handler 中使用torch.get_num_threads来检查线程数。每个 4 个主要工作线程启动了一个物理核心数56的线程总共启动了 56x4 224 个线程这比总核心数 112 多。因此核心肯定会严重重叠逻辑核心利用率高-多个工作线程同时使用多个核心。此外由于线程没有固定到特定的 CPU 核心操作系统会定期将线程调度到位于不同插槽中的核心。
CPU 使用率 启动了 4 个主要工作线程然后每个线程在所有核心上启动了一个物理核心数56的线程包括逻辑核心。
核心绑定停顿 我们观察到 88.4%的非均匀内存访问降低了管道效率。核心绑定停顿表示 CPU 中可用执行单元的使用不佳。例如连续的几个 GEMM 指令竞争由超线程核心共享的融合乘加FMA或点积DP执行单元可能导致核心绑定停顿。正如前一节所述逻辑核心的使用加剧了这个问题。 未填充微操作uOps的空管道槽位被归因于停顿。例如没有核心固定时CPU 使用率可能不会有效地用于计算而是用于来自 Linux 内核的其他操作如线程调度。我们可以看到__sched_yield贡献了大部分自旋时间。
线程迁移
没有核心固定调度程序可能会将在一个核心上执行的线程迁移到另一个核心。线程迁移可能会使线程与已经获取到缓存中的数据分离导致更长的数据访问延迟。在 NUMA 系统中当线程在插座之间迁移时这个问题会加剧。已经获取到本地内存高速缓存的数据现在变成了远程内存速度要慢得多。 通常总线程数应小于或等于核心支持的总线程数。在上面的示例中我们注意到大量线程在 core_51 上执行而不是预期的 2 个线程因为 Intel® Xeon® Platinum 8180 CPU 启用了超线程。这表明线程迁移。 此外请注意线程TID:97097正在大量 CPU 核心上执行表明 CPU 迁移。例如此线程在 cpu_81 上执行然后迁移到 cpu_14然后迁移到 cpu_5依此类推。此外请注意此线程多次在不同插槽之间迁移导致内存访问非常低效。例如此线程在 cpu_70NUMA 节点 0上执行然后迁移到 cpu_100NUMA 节点 1然后迁移到 cpu_24NUMA 节点 0。
非均匀内存访问分析 比较随时间变化的本地与远程内存访问。我们观察到大约一半即 51.09%的内存访问是远程访问表明 NUMA 配置不佳。
2. torch.set_num_threads 物理核心数/工作线程数无核心固定
为了与启动器的核心固定进行苹果对苹果的比较我们将将线程数设置为核心数除以工作线程数启动器在内部执行此操作。在base_handler中添加以下代码片段
torch.set_num_threads(num_physical_cores/num_workers) 与之前一样没有核心固定这些线程没有与特定 CPU 核心关联导致操作系统周期性地在位于不同插座的核心上调度线程。
CPU 使用率 启动了 4 个主要工作线程然后每个线程在所有核心上启动了num_physical_cores/num_workers14个线程包括逻辑核心。
核心绑定停顿 尽管核心绑定停顿的百分比从 88.4%降至 73.5%但核心绑定仍然非常高。 线程迁移 与之前类似没有核心固定时线程TID:94290在大量 CPU 核心上执行表明 CPU 迁移。我们再次注意到跨插槽的线程迁移导致内存访问非常低效。例如此线程在 cpu_78NUMA 节点 0上执行然后迁移到 cpu_108NUMA 节点 1。
非均匀内存访问分析 尽管相比原始的 51.09%有所改善但仍然有 40.45%的内存访问是远程的表明 NUMA 配置不够理想。
3. 启动器核心绑定
启动器将内部将物理核心均匀分配给工作线程并将它们绑定到每个工作线程。提醒一下默认情况下启动器仅使用物理核心。在此示例中启动器将工作线程 0 绑定到核心 0-13NUMA 节点 0工作线程 1 绑定到核心 14-27NUMA 节点 0工作线程 2 绑定到核心 28-41NUMA 节点 1工作线程 3 绑定到核心 42-55NUMA 节点 1。这样做可以确保核心在工作线程之间不重叠并避免逻辑核心的使用。
CPU 使用率 启动了 4 个主要工作线程然后每个工作线程启动了num_physical_cores/num_workers数量14的线程这些线程与分配的物理核心亲和。
核心绑定的停顿 核心绑定的停顿从原始的 88.4%显著减少到 46.2% - 几乎提高了 2 倍。 我们验证了通过核心绑定大部分 CPU 时间有效地用于计算 - 自旋时间为 0.256 秒。
线程迁移 我们验证了 OMP 主线程0 绑定到分配的物理核心42-55并且没有跨插槽迁移。
非均匀内存访问分析 现在几乎所有89.52%内存访问都是本地访问。
结论
在本博客中我们展示了正确设置 CPU 运行时配置如何显著提升开箱即用的 CPU 性能。
我们已经介绍了一些通用的 CPU 性能调优原则和建议 在启用超线程的系统中通过核心绑定将线程亲和性设置为仅在物理核心上避免逻辑核心。 在具有 NUMA 的多插槽系统中通过将线程亲和性设置为特定插槽避免跨插槽的远程内存访问。
我们从基本原理上直观地解释了这些想法并通过性能分析验证了性能提升。最后我们将所有学到的东西应用到 TorchServe 中以提升开箱即用的 TorchServe CPU 性能。
这些原则可以通过一个易于使用的启动脚本自动配置该脚本已经集成到 TorchServe 中。
对于感兴趣的读者请查看以下文档 CPU 特定优化 最大化 Intel®软件优化对 CPU 上 PyTorch*性能的影响 性能调优指南 启动脚本使用指南 自上而下的微体系结构分析方法 为基准测试配置 oneDNN Intel® VTune™ Profiler Intel® VTune™ Profiler 用户指南
请继续关注有关通过Intel® Extension for PyTorch*优化 CPU 内核和高级启动器配置如内存分配器的后续文章。
致谢
我们要感谢 Ashok EmaniIntel和 Jiong GongIntel在这篇博客的许多步骤中给予我们巨大的指导和支持以及详尽的反馈和审查。我们还要感谢 Hamid ShojanazeriMeta、Li NingAWS和 Jing XuIntel在代码审查中提供的有用反馈。以及 Suraj SubramanianMeta和 Geeta ChauhanMeta在博客中提供的有用反馈。
从第一原则开始理解 PyTorch 英特尔 CPU 性能第 2 部分 原文pytorch.org/tutorials/intermediate/torchserve_with_ipex_2.html 译者飞龙 协议CC BY-NC-SA 4.0 作者Min Jean ChoJing XuMark Saroufim
在Grokking PyTorch 英特尔 CPU 性能从第一原则开始教程中我们介绍了如何调整 CPU 运行时配置如何对其进行性能分析以及如何将它们集成到TorchServe中以获得优化的 CPU 性能。
在本教程中我们将通过英特尔® PyTorch*扩展启动器演示如何通过内存分配器提高性能并通过英特尔® PyTorch*扩展在 CPU 上优化内核并将它们应用于 TorchServe展示 ResNet50 的吞吐量提升了 7.71 倍BERT 的吞吐量提升了 2.20 倍。 先决条件
在整个本教程中我们将使用自顶向下的微体系结构分析TMA来对性能进行分析并展示未经优化或未调整的深度学习工作负载通常主要受到后端受限内存受限、核心受限的影响并通过英特尔® PyTorch*扩展演示优化技术以改善后端受限。我们将使用toplev这是pmu-tools的一部分工具构建在Linux perf之上用于 TMA。
我们还将使用英特尔® VTune™ Profiler 的仪器化和跟踪技术ITT以更精细的粒度进行性能分析。
自顶向下的微体系结构分析方法TMA
在调整 CPU 以获得最佳性能时了解瓶颈所在是很有用的。大多数 CPU 核心都有芯片上的性能监控单元PMUs。PMUs 是 CPU 核心内的专用逻辑单元用于计算系统上发生的特定硬件事件。这些事件的示例可能是缓存未命中或分支误预测。PMUs 用于自顶向下的微体系结构分析TMA以识别瓶颈。TMA 包括如下层次结构 顶层即一级指标收集退休、错误猜测、前端受限、后端受限。CPU 的流水线在概念上可以简化并分为两部分前端和后端。前端负责获取程序代码并将其解码为称为微操作uOps的低级硬件操作。然后将 uOps 传送到后端这个过程称为分配。一旦分配后端负责在可用的执行单元中执行 uOp。uOp 的执行完成称为退休。相反错误猜测是指在退休之前取消了被推测获取的 uOps例如在误预测分支的情况下。这些指标中的每一个都可以进一步细分为后续级别以确定瓶颈。
调整后端受限
大多数未调整的深度学习工作负载将是后端受限的。解决后端受限通常是解决导致退休时间超过必要时间的延迟源。如上所示后端受限有两个子指标 - 核心受限和内存受限。
内存绑定停顿与内存子系统相关。例如最后一级缓存LLC 或 L3 缓存缺失导致访问 DRAM。扩展深度学习模型通常需要大量计算。高计算利用率要求在执行单元需要执行 uOps 时数据可用。这需要预取数据并在缓存中重用数据而不是多次从主存储器中获取相同的数据这会导致执行单元在数据返回时被饿死。在整个教程中我们将展示更高效的内存分配器、操作融合、内存布局格式优化如何减少内存绑定的开销提高缓存局部性。
核心绑定停顿表示可用执行单元的子优化使用而没有未完成的内存访问。例如连续的多个通用矩阵乘法GEMM指令竞争融合乘加FMA或点积DP执行单元可能导致核心绑定停顿。关键的深度学习核心包括 DP 核心已经通过oneDNN 库oneAPI 深度神经网络库进行了优化减少了核心绑定的开销。
GEMM、卷积、反卷积等操作是计算密集型的。而池化、批量归一化、激活函数如 ReLU 等操作是内存绑定的。
Intel® VTune™ Profiler 的仪器化和跟踪技术ITT
Intel® VTune Profiler 的 ITT API 是一个有用的工具用于注释您的工作负载的区域以便以更细粒度的注释-OP/函数/子函数粒度进行跟踪和可视化。通过在 PyTorch 模型的 OP 级别进行注释Intel® VTune Profiler 的 ITT 实现了 OP 级别的性能分析。Intel® VTune Profiler 的 ITT 已经集成到PyTorch Autograd Profiler中。¹
该功能必须通过*with torch.autograd.profiler.emit_itt()*来显式启用。
带有 Intel® Extension for PyTorch*的 TorchServe
Intel® Extension for PyTorch*是一个 Python 包用于通过在 Intel 硬件上进行额外性能优化来扩展 PyTorch。
Intel® Extension for PyTorch已经集成到 TorchServe 中以提高性能。² 对于自定义处理程序脚本我们建议添加intel_extension_for_pytorch*包。
该功能必须通过在config.properties中设置ipex_enabletrue来显式启用。
在本节中我们将展示后端绑定通常是未经优化或未调整的深度学习工作负载的主要瓶颈并通过 Intel® Extension for PyTorch演示优化技术以改善后端绑定后端绑定有两个子指标-内存绑定和核心绑定。更高效的内存分配器、操作融合、内存布局格式优化改善了内存绑定。理想情况下通过优化操作符和更好的缓存局部性内存绑定可以改善为核心绑定。关键的深度学习基元如卷积、矩阵乘法、点积已经通过 Intel® Extension for PyTorch和 oneDNN 库进行了优化改善了核心绑定。
利用高级启动器配置内存分配器
从性能的角度来看内存分配器起着重要作用。更高效的内存使用减少了不必要的内存分配或销毁的开销从而实现更快的执行。在实践中的深度学习工作负载中特别是在像 TorchServe 这样的大型多核系统或服务器上运行的工作负载中TCMalloc 或 JeMalloc 通常比默认的 PyTorch 内存分配器 PTMalloc 具有更好的内存使用。
TCMallocJeMallocPTMalloc
TCMalloc 和 JeMalloc 都使用线程本地缓存来减少线程同步的开销并通过使用自旋锁和每个线程的竞技场来减少锁争用。TCMalloc 和 JeMalloc 减少了不必要的内存分配和释放的开销。两个分配器通过大小对内存分配进行分类以减少内存碎片化的开销。
使用启动器用户可以通过选择三个启动器旋钮之一来轻松尝试不同的内存分配器 –enable_tcmallocTCMalloc、–enable_jemallocJeMalloc、–use_default_allocatorPTMalloc。
练习
让我们对比 PTMalloc 和 JeMalloc 进行分析。
我们将使用启动器指定内存分配器并将工作负载绑定到第一个插槽的物理核心以避免任何 NUMA 复杂性 – 仅对内存分配器的影响进行分析。
以下示例测量了 ResNet50 的平均推理时间
import torch
import torchvision.models as models
import timemodel models.resnet50(pretrainedFalse)
model.eval()
batch_size 32
data torch.rand(batch_size, 3, 224, 224)# warm up
for _ in range(100):model(data)# measure
# Intel® VTune Profilers ITT context manager
with torch.autograd.profiler.emit_itt():start time.time()for i in range(100):# Intel® VTune Profilers ITT to annotate each steptorch.profiler.itt.range_push(step_{}.format(i))model(data)torch.profiler.itt.range_pop()end time.time()print(Inference took {:.2f} ms in average.format((end-start)/100*1000)) 让我们收集一级 TMA 指标。 一级 TMA 显示 PTMalloc 和 JeMalloc 都受后端限制。超过一半的执行时间被后端阻塞。让我们再深入一层。 二级 TMA 显示后端受限是由内存受限引起的。让我们再深入一层。 大多数内存受限指标确定了从 L1 缓存到主存储器的内存层次结构中的瓶颈。在给定级别上受限的热点表明大部分数据是从该缓存或内存级别检索的。优化应该专注于将数据移动到核心附近。三级 TMA 显示 PTMalloc 受 DRAM Bound 限制。另一方面JeMalloc 受 L1 Bound 限制 – JeMalloc 将数据移动到核心附近从而实现更快的执行。
让我们看看 Intel® VTune Profiler ITT 跟踪。在示例脚本中我们已经注释了推理循环的每个 step_x。 时间轴图中跟踪了每个步骤。在最后一步step_99的模型推理持续时间从 304.308 毫秒减少到 261.843 毫秒。
使用 TorchServe 进行练习
让我们使用 TorchServe 对比 PTMalloc 和 JeMalloc 进行分析。
我们将使用 TorchServe apache-bench 基准测试 进行 ResNet50 FP32、批量大小 32、并发数 32、请求数 8960. 其他所有参数与 默认参数 相同。
与之前的练习一样我们将使用启动器指定内存分配器并将工作负载绑定到第一个插槽的物理核心。为此用户只需在 config.properties 中添加几行即可
PTMalloc
cpu_launcher_enabletrue
cpu_launcher_args--node_id 0 --use_default_allocator JeMalloc
cpu_launcher_enabletrue
cpu_launcher_args--node_id 0 --enable_jemalloc 让我们收集一级 TMA 指标。 让我们再深入一层。 让我们使用 Intel® VTune Profiler ITT 对 TorchServe 推理范围 进行注释以便以推理级别的粒度进行分析。由于 TorchServe 架构 包括几个子组件包括用于处理请求/响应的 Java 前端和用于在模型上运行实际推理的 Python 后端因此使用 Intel® VTune Profiler ITT 限制在推理级别收集跟踪数据是有帮助的。 每个推断调用都在时间线图中被跟踪。最后一个模型推断的持续时间从 561.688 毫秒减少到 251.287 毫秒 - 加速 2.2 倍。 时间线图可以展开以查看操作级别的性能分析结果。aten::conv2d的持续时间从 16.401 毫秒减少到 6.392 毫秒 - 加速 2.6 倍。
在本节中我们已经证明 JeMalloc 可以比默认的 PyTorch 内存分配器 PTMalloc 提供更好的性能有效的线程本地缓存可以改善后端绑定。
Intel® Extension for PyTorch*
三个主要的Intel® Extension for PyTorch*优化技术运算符、图形、运行时如下所示
Intel® Extension for PyTorch* 优化技术运算符
| 矢量化和多线程 低精度 BF16/INT8 计算 为了更好的缓存局部性进行的数据布局优化
| 常量折叠以减少计算 为了更好的缓存局部性而进行的操作融合
| 线程亲和性 内存缓冲池 GPU 运行时 启动器
|
运算符优化
优化的运算符和内核通过 PyTorch 调度机制进行注册。这些运算符和内核从英特尔硬件的本机矢量化特性和矩阵计算特性加速。在执行过程中Intel® Extension for PyTorch拦截 ATen 运算符的调用并用这些优化的运算符替换原始运算符。像卷积、线性等流行的运算符已经在 Intel® Extension for PyTorch中进行了优化。
练习
让我们使用 Intel® Extension for PyTorch*对优化的运算符进行性能分析。我们将比较在代码中添加和不添加这些行的变化。
与以前的练习一样我们将将工作负载绑定到第一个插槽的物理核心。
import torchclass Model(torch.nn.Module):def __init__(self):super(Model, self).__init__()self.conv torch.nn.Conv2d(16, 33, 3, stride2)self.relu torch.nn.ReLU()def forward(self, x):x self.conv(x)x self.relu(x)return xmodel Model()
model.eval()
data torch.rand(20, 16, 50, 100)#################### code changes ####################
import intel_extension_for_pytorch as ipex
model ipex.optimize(model)
######################################################print(model) 该模型由两个操作组成——Conv2d 和 ReLU。通过打印模型对象我们得到以下输出。 让我们收集一级 TMA 指标。 注意后端绑定从 68.9 减少到 38.5 - 加速 1.8 倍。
此外让我们使用 PyTorch Profiler 进行性能分析。 注意 CPU 时间从 851 微秒减少到 310 微秒 - 加速 2.7 倍。
图形优化
强烈建议用户利用 Intel® Extension for PyTorch与TorchScript进一步优化图形。为了通过 TorchScript 进一步优化性能Intel® Extension for PyTorch支持常用 FP32/BF16 运算符模式的 oneDNN 融合如 Conv2DReLU、LinearReLU 等以减少运算符/内核调用开销提高缓存局部性。一些运算符融合允许保持临时计算、数据类型转换、数据布局以提高缓存局部性。此外对于 INT8Intel® Extension for PyTorch*具有内置的量化配方为包括 CNN、NLP 和推荐模型在内的流行 DL 工作负载提供良好的统计精度。量化模型然后通过 oneDNN 融合支持进行优化。
练习
让我们使用 TorchScript 对 FP32 图形优化进行性能分析。
与以前的练习一样我们将将工作负载绑定到第一个插槽的物理核心。
import torchclass Model(torch.nn.Module):def __init__(self):super(Model, self).__init__()self.conv torch.nn.Conv2d(16, 33, 3, stride2)self.relu torch.nn.ReLU()def forward(self, x):x self.conv(x)x self.relu(x)return xmodel Model()
model.eval()
data torch.rand(20, 16, 50, 100)#################### code changes ####################
import intel_extension_for_pytorch as ipex
model ipex.optimize(model)
####################################################### torchscript
with torch.no_grad():model torch.jit.trace(model, data)model torch.jit.freeze(model) 让我们收集一级 TMA 指标。 注意后端绑定从 67.1 减少到 37.5 - 加速 1.8 倍。
此外让我们使用 PyTorch Profiler 进行性能分析。 请注意使用 Intel® PyTorch 扩展Conv ReLU 操作符被融合CPU 时间从 803 微秒减少到 248 微秒加速了 3.2 倍。oneDNN eltwise 后操作使得可以将一个原语与一个逐元素原语融合。这是最流行的融合类型之一一个 eltwise通常是激活函数如 ReLU与前面的卷积或内积。请查看下一节中显示的 oneDNN 详细日志。
Channels Last 内存格式
在模型上调用ipex.optimize时Intel® PyTorch 扩展会自动将模型转换为优化的内存格式即 channels last。Channels last 是一种更适合 Intel 架构的内存格式。与 PyTorch 默认的 channels first NCHWbatch, channels, height, width内存格式相比channels last NHWCbatch, height, width, channels内存格式通常可以加速卷积神经网络具有更好的缓存局部性。
需要注意的一点是转换内存格式是昂贵的。因此最好在部署之前将内存格式转换一次并在部署过程中尽量减少内存格式转换。当数据通过模型的层传播时最后通道的内存格式会通过连续的支持最后通道的层例如Conv2d - ReLU - Conv2d保持不变转换仅在不支持最后通道的层之间进行。有关更多详细信息请参阅内存格式传播。
练习
让我们演示最后通道优化。
import torchclass Model(torch.nn.Module):def __init__(self):super(Model, self).__init__()self.conv torch.nn.Conv2d(16, 33, 3, stride2)self.relu torch.nn.ReLU()def forward(self, x):x self.conv(x)x self.relu(x)return xmodel Model()
model.eval()
data torch.rand(20, 16, 50, 100)import intel_extension_for_pytorch as ipex
############################### code changes ###############################
ipex.disable_auto_channels_last() # omit this line for channels_last (default)
############################################################################
model ipex.optimize(model)with torch.no_grad():model torch.jit.trace(model, data)model torch.jit.freeze(model) 我们将使用oneDNN 详细模式这是一个帮助收集有关 oneDNN 图级别信息的工具例如操作融合、执行 oneDNN 原语所花费的内核执行时间。有关更多信息请参考oneDNN 文档。 以上是来自通道首的 oneDNN 详细信息。我们可以验证从权重和数据进行重新排序然后进行计算最后将输出重新排序。 以上是来自通道最后的 oneDNN 详细信息。我们可以验证通道最后的内存格式避免了不必要的重新排序。
使用 Intel® PyTorch 扩展提升性能
以下总结了 TorchServe 与 Intel® Extension for PyTorch*在 ResNet50 和 BERT-base-uncased 上的性能提升。 使用 TorchServe 进行练习
让我们使用 TorchServe 来分析 Intel® Extension for PyTorch*的优化。
我们将使用 ResNet50 FP32 TorchScript 的TorchServe apache-bench 基准测试批量大小为 32并发数为 32请求为 8960。所有其他参数与默认参数相同。
与上一个练习一样我们将使用启动器将工作负载绑定到第一个插槽的物理核心。为此用户只需在config.properties中添加几行代码
cpu_launcher_enabletrue
cpu_launcher_args--node_id 0 让我们收集一级 TMA 指标。 Level-1 TMA 显示两者都受到后端的限制。正如之前讨论的大多数未调整的深度学习工作负载将受到后端的限制。注意后端限制从 70.0 降至 54.1。让我们再深入一层。 如前所述后端绑定有两个子指标 - 内存绑定和核心绑定。内存绑定表示工作负载未经优化或未充分利用理想情况下内存绑定操作可以通过优化 OPs 和改善缓存局部性来改善为核心绑定。Level-2 TMA 显示后端绑定从内存绑定改善为核心绑定。让我们深入一层。 在像 TorchServe 这样的模型服务框架上为生产扩展深度学习模型需要高计算利用率。这要求数据通过预取并在执行单元需要执行 uOps 时在缓存中重复使用。Level-3 TMA 显示后端内存绑定从 DRAM 绑定改善为核心绑定。
与 TorchServe 之前的练习一样让我们使用 Intel® VTune Profiler ITT 来注释TorchServe 推断范围以便以推断级别的粒度进行分析。 时间轴图中跟踪了每个推断调用。最后一个推断调用的持续时间从 215.731 毫秒减少到 95.634 毫秒 - 2.3 倍加速。 时间轴图可以展开以查看操作级别的分析结果。请注意Conv ReLU 已经被融合持续时间从 6.393 毫秒1.731 毫秒减少到 3.408 毫秒 - 2.4 倍加速。
结论
在本教程中我们使用了自顶向下微架构分析TMA和 Intel® VTune™ Profiler 的仪器化和跟踪技术ITT来演示 通常未经优化或调整不足的深度学习工作负载的主要瓶颈是后端绑定它有两个子指标即内存绑定和核心绑定。 Intel® PyTorch*的更高效的内存分配器、操作融合、内存布局格式优化改善了内存绑定。 关键的深度学习基元如卷积、矩阵乘法、点积等已经被 Intel® PyTorch*扩展和 oneDNN 库进行了优化提高了核心绑定。 Intel® PyTorch*扩展已经集成到 TorchServe 中具有易于使用的 API。 TorchServe 与 Intel® PyTorch*扩展展示了 ResNet50 的 7.71 倍吞吐量提升以及 BERT 的 2.20 倍吞吐量提升。
相关阅读
自顶向下微架构分析方法
自顶向下性能分析方法
使用 Intel® PyTorch*扩展加速 PyTorch
致谢
我们要感谢 Ashok EmaniIntel和 Jiong GongIntel在本教程的许多步骤中提供的巨大指导和支持以及全面的反馈和审查。我们还要感谢 Hamid ShojanazeriMeta和 Li NingAWS在代码审查和教程中提供的有用反馈。
开始 - 用 nvFuser 加速您的脚本 原文pytorch.org/tutorials/intermediate/nvfuser_intro_tutorial.html 译者飞龙 协议CC BY-NC-SA 4.0 此教程已被弃用。3 秒后将重定向到主页…
使用 Ax 进行多目标 NAS 原文pytorch.org/tutorials/intermediate/ax_multiobjective_nas_tutorial.html 译者飞龙 协议CC BY-NC-SA 4.0 注意
点击这里下载完整示例代码
作者 David Eriksson, Max Balandat以及 Meta 的自适应实验团队。
在本教程中我们展示如何使用Ax在流行的 MNIST 数据集上运行简单神经网络模型的多目标神经架构搜索NAS。虽然潜在的方法通常用于更复杂的模型和更大的数据集但我们选择了一个在笔记本电脑上可以轻松运行的教程不到 20 分钟即可完成。
在许多 NAS 应用中存在着多个感兴趣目标之间的自然权衡。例如在部署模型到设备上时我们可能希望最大化模型性能例如准确性同时最小化竞争指标如功耗、推理延迟或模型大小以满足部署约束。通常情况下通过接受略低的模型性能我们可以大大减少预测的计算需求或延迟。探索这种权衡的原则方法是可扩展和可持续人工智能的关键推动因素并在 Meta 上有许多成功的应用案例 - 例如查看我们关于自然语言理解模型的案例研究。
在我们的示例中我们将调整两个隐藏层的宽度、学习率、dropout 概率、批量大小和训练周期数。目标是在性能验证集上的准确率和模型大小模型参数数量之间进行权衡。
本教程使用以下 PyTorch 库 PyTorch Lightning指定模型和训练循环 TorchX用于远程/异步运行训练作业 BoTorch为 Ax 的算法提供动力的贝叶斯优化库
定义 TorchX 应用
我们的目标是优化在mnist_train_nas.py中定义的 PyTorch Lightning 训练作业。为了使用 TorchX 实现这一目标我们编写了一个辅助函数该函数接受训练作业的架构和超参数的值并创建一个具有适当设置的TorchX AppDef。
from pathlib import Pathimport torchxfrom torchx import specs
from torchx.components import utilsdef trainer(log_path: str,hidden_size_1: int,hidden_size_2: int,learning_rate: float,epochs: int,dropout: float,batch_size: int,trial_idx: int -1,
) - specs.AppDef:# define the log path so we can pass it to the TorchX AppDefif trial_idx 0:log_path Path(log_path).joinpath(str(trial_idx)).absolute().as_posix()return utils.python(# command line arguments to the training script--log_path,log_path,--hidden_size_1,str(hidden_size_1),--hidden_size_2,str(hidden_size_2),--learning_rate,str(learning_rate),--epochs,str(epochs),--dropout,str(dropout),--batch_size,str(batch_size),# other config optionsnametrainer,scriptmnist_train_nas.py,imagetorchx.version.TORCHX_IMAGE,) 设置 Runner
Ax 的Runner抽象允许编写与各种后端的接口。Ax 已经为 TorchX 提供了 Runner因此我们只需要配置它。在本教程中我们以完全异步的方式在本地运行作业。
为了在集群上启动它们您可以指定一个不同的 TorchX 调度程序并相应地调整配置。例如如果您有一个 Kubernetes 集群您只需要将调度程序从local_cwd更改为kubernetes。
import tempfile
from ax.runners.torchx import TorchXRunner# Make a temporary dir to log our results into
log_dir tempfile.mkdtemp()ax_runner TorchXRunner(tracker_base/tmp/,componenttrainer,# NOTE: To launch this job on a cluster instead of locally you can# specify a different scheduler and adjust arguments appropriately.schedulerlocal_cwd,component_const_params{log_path: log_dir},cfg{},
) 设置SearchSpace
首先我们定义我们的搜索空间。Ax 支持整数和浮点类型的范围参数也支持选择参数可以具有非数字类型如字符串。我们将调整隐藏层大小、学习率、丢失率和时代数作为范围参数并将批量大小调整为有序选择参数以强制其为 2 的幂。
from ax.core import (ChoiceParameter,ParameterType,RangeParameter,SearchSpace,
)parameters [# NOTE: In a real-world setting, hidden_size_1 and hidden_size_2# should probably be powers of 2, but in our simple example this# would mean that num_params cant take on that many values, which# in turn makes the Pareto frontier look pretty weird.RangeParameter(namehidden_size_1,lower16,upper128,parameter_typeParameterType.INT,log_scaleTrue,),RangeParameter(namehidden_size_2,lower16,upper128,parameter_typeParameterType.INT,log_scaleTrue,),RangeParameter(namelearning_rate,lower1e-4,upper1e-2,parameter_typeParameterType.FLOAT,log_scaleTrue,),RangeParameter(nameepochs,lower1,upper4,parameter_typeParameterType.INT,),RangeParameter(namedropout,lower0.0,upper0.5,parameter_typeParameterType.FLOAT,),ChoiceParameter( # NOTE: ChoiceParameters dont require log-scalenamebatch_size,values[32, 64, 128, 256],parameter_typeParameterType.INT,is_orderedTrue,sort_valuesTrue,),
]search_space SearchSpace(parametersparameters,# NOTE: In practice, it may make sense to add a constraint# hidden_size_2 hidden_size_1parameter_constraints[],
) 设置度量
Ax 有一个度量的概念它定义了结果的属性以及如何获取这些结果的观察。这允许例如编码数据如何从某个分布式执行后端获取并在传递给 Ax 之前进行后处理。
在本教程中我们将使用多目标优化来最大化验证准确性并最小化模型参数数量。后者代表了模型延迟的简单代理对于小型机器学习模型来说很难准确估计在实际应用中我们会在设备上运行模型时对延迟进行基准测试。
在我们的示例中TorchX 将以完全异步的方式在本地运行训练作业并根据试验索引参见上面的trainer()函数将结果写入log_dir。我们将定义一个度量类该类知道该日志目录。通过子类化TensorboardCurveMetric我们可以免费获得读取和解析 TensorBoard 日志的逻辑。
from ax.metrics.tensorboard import TensorboardCurveMetricclass MyTensorboardMetric(TensorboardCurveMetric):# NOTE: We need to tell the new TensorBoard metric how to get the id /# file handle for the TensorBoard logs from a trial. In this case# our convention is to just save a separate file per trial in# the prespecified log dir.classmethoddef get_ids_from_trials(cls, trials):return {trial.index: Path(log_dir).joinpath(str(trial.index)).as_posix()for trial in trials}# This indicates whether the metric is queryable while the trial is# still running. We dont use this in the current tutorial, but Ax# utilizes this to implement trial-level early-stopping functionality.classmethoddef is_available_while_running(cls):return False 现在我们可以实例化准确率和模型参数数量的指标。这里 curve_name 是 TensorBoard 日志中指标的名称而 name 是 Ax 内部使用的指标名称。我们还指定 lower_is_better 来指示这两个指标的有利方向。
val_acc MyTensorboardMetric(nameval_acc,curve_nameval_acc,lower_is_betterFalse,
)
model_num_params MyTensorboardMetric(namenum_params,curve_namenum_params,lower_is_betterTrue,
) 设置OptimizationConfig
告诉 Ax 应该优化的方式是通过OptimizationConfig。在这里我们使用MultiObjectiveOptimizationConfig因为我们将执行多目标优化。
此外Ax 支持通过指定目标阈值对不同指标设置约束这些约束限制了我们想要探索的结果空间的区域。在本例中我们将约束验证准确率至少为 0.9494%模型参数数量最多为 80,000。
from ax.core import MultiObjective, Objective, ObjectiveThreshold
from ax.core.optimization_config import MultiObjectiveOptimizationConfigopt_config MultiObjectiveOptimizationConfig(objectiveMultiObjective(objectives[Objective(metricval_acc, minimizeFalse),Objective(metricmodel_num_params, minimizeTrue),],),objective_thresholds[ObjectiveThreshold(metricval_acc, bound0.94, relativeFalse),ObjectiveThreshold(metricmodel_num_params, bound80_000, relativeFalse),],
) 创建 Ax 实验
在 Ax 中Experiment 对象是存储有关问题设置的所有信息的对象。
from ax.core import Experimentexperiment Experiment(nametorchx_mnist,search_spacesearch_space,optimization_configopt_config,runnerax_runner,
) 选择生成策略
GenerationStrategy 是我们希望执行优化的抽象表示。虽然这可以定制如果您愿意这样做请参阅此教程但在大多数情况下Ax 可以根据搜索空间、优化配置和我们想要运行的总试验次数自动确定适当的策略。
通常Ax 选择在开始基于模型的贝叶斯优化策略之前评估一些随机配置。
total_trials 48 # total evaluation budgetfrom ax.modelbridge.dispatch_utils import choose_generation_strategygs choose_generation_strategy(search_spaceexperiment.search_space,optimization_configexperiment.optimization_config,num_trialstotal_trials,) [INFO 02-03 05:14:14] ax.modelbridge.dispatch_utils: Using Models.BOTORCH_MODULAR since there are more ordered parameters than there are categories for the unordered categorical parameters.
[INFO 02-03 05:14:14] ax.modelbridge.dispatch_utils: Calculating the number of remaining initialization trials based on num_initialization_trialsNone max_initialization_trialsNone num_tunable_parameters6 num_trials48 use_batch_trialsFalse
[INFO 02-03 05:14:14] ax.modelbridge.dispatch_utils: calculated num_initialization_trials9
[INFO 02-03 05:14:14] ax.modelbridge.dispatch_utils: num_completed_initialization_trials0 num_remaining_initialization_trials9
[INFO 02-03 05:14:14] ax.modelbridge.dispatch_utils: verbose, disable_progbar, and jit_compile are not yet supported when using choose_generation_strategy with ModularBoTorchModel, dropping these arguments.
[INFO 02-03 05:14:14] ax.modelbridge.dispatch_utils: Using Bayesian Optimization generation strategy: GenerationStrategy(nameSobolBoTorch, steps[Sobol for 9 trials, BoTorch for subsequent trials]). Iterations after 9 will take longer to generate due to model-fitting. 配置调度程序
Scheduler 充当优化的循环控制器。它与后端通信启动试验检查它们的状态并检索结果。在本教程中它只是读取和解析本地保存的日志。在远程执行设置中它将调用 API。来自 Ax Scheduler 教程 的以下插图总结了 Scheduler 与用于运行试验评估的外部系统的交互方式 调度程序 需要 实验 和 生成策略。一组选项可以通过 调度程序选项 传递进来。在这里我们配置了总评估次数以及 max_pending_trials即应同时运行的最大试验数。在我们的本地设置中这是作为单独进程运行的训练作业的数量而在远程执行设置中这将是您想要并行使用的机器数量。
from ax.service.scheduler import Scheduler, SchedulerOptionsscheduler Scheduler(experimentexperiment,generation_strategygs,optionsSchedulerOptions(total_trialstotal_trials, max_pending_trials4),
) [INFO 02-03 05:14:15] Scheduler: Scheduler requires experiment to have immutable search space and optimization config. Setting property immutable_search_space_and_opt_config to True on experiment. 运行优化
现在一切都配置好了我们可以让 Ax 以完全自动化的方式运行优化。调度程序将定期检查日志以获取所有当前运行试验的状态如果一个试验完成调度程序将更新其在实验中的状态并获取贝叶斯优化算法所需的观察结果。
scheduler.run_all_trials() [INFO 02-03 05:14:15] Scheduler: Running trials [0]...
[INFO 02-03 05:14:16] Scheduler: Running trials [1]...
[INFO 02-03 05:14:17] Scheduler: Running trials [2]...
[INFO 02-03 05:14:18] Scheduler: Running trials [3]...
[INFO 02-03 05:14:19] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 05:14:20] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 05:14:21] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 05:14:23] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 02-03 05:14:27] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 02-03 05:14:32] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 02-03 05:14:39] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 4).
[INFO 02-03 05:14:51] Scheduler: Waiting for completed trials (for 17 sec, currently running trials: 4).
[INFO 02-03 05:15:08] Scheduler: Retrieved COMPLETED trials: [1, 3].
[INFO 02-03 05:15:08] Scheduler: Fetching data for trials: [1, 3].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:15:08] Scheduler: Running trials [4]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:15:09] Scheduler: Running trials [5]...
[INFO 02-03 05:15:10] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 05:15:11] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 05:15:13] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 05:15:15] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 02-03 05:15:18] Scheduler: Retrieved COMPLETED trials: [2].
[INFO 02-03 05:15:18] Scheduler: Fetching data for trials: [2].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:15:18] Scheduler: Running trials [6]...
[INFO 02-03 05:15:19] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 05:15:20] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 05:15:22] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 05:15:24] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 02-03 05:15:27] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 02-03 05:15:33] Scheduler: Retrieved COMPLETED trials: [0].
[INFO 02-03 05:15:33] Scheduler: Fetching data for trials: [0].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:15:33] Scheduler: Running trials [7]...
[INFO 02-03 05:15:34] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 05:15:35] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 05:15:36] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 05:15:38] Scheduler: Retrieved COMPLETED trials: [4].
[INFO 02-03 05:15:38] Scheduler: Fetching data for trials: [4].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:15:38] Scheduler: Running trials [8]...
[INFO 02-03 05:15:39] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 05:15:40] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 05:15:42] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 05:15:44] Scheduler: Retrieved COMPLETED trials: [5].
[INFO 02-03 05:15:44] Scheduler: Fetching data for trials: [5].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:15:46] Scheduler: Running trials [9]...
[INFO 02-03 05:15:47] Scheduler: Retrieved COMPLETED trials: [6].
[INFO 02-03 05:15:47] Scheduler: Fetching data for trials: [6].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:15:50] Scheduler: Running trials [10]...
[INFO 02-03 05:15:51] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 05:15:52] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 05:15:54] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 05:15:56] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 02-03 05:15:59] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 02-03 05:16:05] Scheduler: Retrieved COMPLETED trials: [8].
[INFO 02-03 05:16:05] Scheduler: Fetching data for trials: [8].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:16:06] Scheduler: Running trials [11]...
[INFO 02-03 05:16:07] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 4).
[INFO 02-03 05:16:08] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 4).
[INFO 02-03 05:16:10] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 4).
[INFO 02-03 05:16:12] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 4).
[INFO 02-03 05:16:16] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 4).
[INFO 02-03 05:16:21] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 4).
[INFO 02-03 05:16:28] Scheduler: Retrieved COMPLETED trials: [7].
[INFO 02-03 05:16:28] Scheduler: Fetching data for trials: [7].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:16:28] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:16:29] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:16:31] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:16:33] Scheduler: Retrieved COMPLETED trials: [9].
[INFO 02-03 05:16:33] Scheduler: Fetching data for trials: [9].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:16:35] Scheduler: Running trials [12]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:16:36] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:16:36] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:16:36] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:16:37] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:16:39] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:16:41] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:16:45] Scheduler: Retrieved COMPLETED trials: [10].
[INFO 02-03 05:16:45] Scheduler: Fetching data for trials: [10].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:16:48] Scheduler: Running trials [13]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:16:48] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:16:48] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:16:48] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:16:49] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:16:50] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:16:52] Scheduler: Retrieved COMPLETED trials: [11].
[INFO 02-03 05:16:52] Scheduler: Fetching data for trials: [11].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:16:55] Scheduler: Running trials [14]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:16:56] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:16:56] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:16:56] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:16:57] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:16:58] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:17:00] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:17:04] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:17:09] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:17:17] Scheduler: Retrieved COMPLETED trials: [12].
[INFO 02-03 05:17:17] Scheduler: Fetching data for trials: [12].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:17:20] Scheduler: Running trials [15]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:17:21] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:17:21] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:17:21] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:17:22] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:17:23] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:17:25] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:17:29] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:17:34] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:17:41] Scheduler: Retrieved COMPLETED trials: [14].
[INFO 02-03 05:17:41] Scheduler: Fetching data for trials: [14].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:17:44] Scheduler: Running trials [16]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:17:45] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:17:45] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:17:45] Scheduler: Retrieved COMPLETED trials: [13].
[INFO 02-03 05:17:45] Scheduler: Fetching data for trials: [13].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:17:48] Scheduler: Running trials [17]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:17:49] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:17:49] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:17:49] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:17:50] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:17:52] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:17:54] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:17:58] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:18:03] Scheduler: Retrieved COMPLETED trials: [15].
[INFO 02-03 05:18:03] Scheduler: Fetching data for trials: [15].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:18:08] Scheduler: Running trials [18]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:18:09] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:18:09] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:18:09] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:18:10] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:18:12] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:18:14] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:18:18] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:18:23] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:18:30] Scheduler: Retrieved COMPLETED trials: [17].
[INFO 02-03 05:18:30] Scheduler: Fetching data for trials: [17].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:18:34] Scheduler: Running trials [19]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:18:35] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:18:35] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:18:35] Scheduler: Retrieved COMPLETED trials: [16].
[INFO 02-03 05:18:35] Scheduler: Fetching data for trials: [16].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:18:39] Scheduler: Running trials [20]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:18:40] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:18:40] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:18:40] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:18:41] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:18:43] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:18:45] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:18:48] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:18:54] Scheduler: Retrieved COMPLETED trials: [18].
[INFO 02-03 05:18:54] Scheduler: Fetching data for trials: [18].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:18:57] Scheduler: Running trials [21]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:18:58] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:18:58] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:18:58] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:18:59] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:19:00] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:19:02] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:19:06] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:19:11] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:19:19] Scheduler: Retrieved COMPLETED trials: [19].
[INFO 02-03 05:19:19] Scheduler: Fetching data for trials: [19].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:19:22] Scheduler: Running trials [22]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:19:23] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:19:23] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:19:23] Scheduler: Retrieved COMPLETED trials: [20].
[INFO 02-03 05:19:23] Scheduler: Fetching data for trials: [20].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:19:26] Scheduler: Running trials [23]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:19:27] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:19:27] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:19:27] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:19:28] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:19:30] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:19:32] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:19:36] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:19:41] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:19:48] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 05:20:00] Scheduler: Retrieved COMPLETED trials: [21, 23].
[INFO 02-03 05:20:00] Scheduler: Fetching data for trials: [21, 23].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:20:03] Scheduler: Running trials [24]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:20:04] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:20:07] Scheduler: Running trials [25]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:20:08] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:20:08] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:20:08] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:20:09] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:20:11] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:20:13] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:20:17] Scheduler: Retrieved COMPLETED trials: [22].
[INFO 02-03 05:20:17] Scheduler: Fetching data for trials: [22].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:20:20] Scheduler: Running trials [26]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:20:21] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:20:21] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:20:21] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:20:22] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:20:24] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:20:26] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:20:29] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:20:34] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:20:42] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 05:20:53] Scheduler: Waiting for completed trials (for 17 sec, currently running trials: 3).
[INFO 02-03 05:21:11] Scheduler: Retrieved COMPLETED trials: 24 - 25.
[INFO 02-03 05:21:11] Scheduler: Fetching data for trials: 24 - 25.
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:21:15] Scheduler: Running trials [27]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:21:16] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:21:19] Scheduler: Running trials [28]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:21:20] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:21:20] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:21:20] Scheduler: Retrieved COMPLETED trials: [26].
[INFO 02-03 05:21:20] Scheduler: Fetching data for trials: [26].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:21:25] Scheduler: Running trials [29]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:21:26] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:21:26] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:21:26] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:21:27] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:21:29] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:21:31] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:21:34] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:21:39] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:21:47] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 05:21:58] Scheduler: Waiting for completed trials (for 17 sec, currently running trials: 3).
[INFO 02-03 05:22:16] Scheduler: Retrieved COMPLETED trials: 27 - 29.
[INFO 02-03 05:22:16] Scheduler: Fetching data for trials: 27 - 29.
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:22:20] Scheduler: Running trials [30]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:22:21] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:22:24] Scheduler: Running trials [31]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:22:25] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:22:30] Scheduler: Running trials [32]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:22:31] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:22:31] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:22:31] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:22:32] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:22:34] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:22:36] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:22:40] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:22:45] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:22:52] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 05:23:04] Scheduler: Waiting for completed trials (for 17 sec, currently running trials: 3).
[INFO 02-03 05:23:21] Scheduler: Retrieved COMPLETED trials: 30 - 32.
[INFO 02-03 05:23:21] Scheduler: Fetching data for trials: 30 - 32.
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:23:26] Scheduler: Running trials [33]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:23:27] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:23:32] Scheduler: Running trials [34]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:23:33] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:23:38] Scheduler: Running trials [35]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:23:38] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:23:38] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:23:38] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:23:39] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:23:41] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:23:43] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:23:47] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:23:52] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:23:59] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 05:24:11] Scheduler: Waiting for completed trials (for 17 sec, currently running trials: 3).
[INFO 02-03 05:24:28] Scheduler: Retrieved COMPLETED trials: 33 - 34.
[INFO 02-03 05:24:28] Scheduler: Fetching data for trials: 33 - 34.
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:24:34] Scheduler: Running trials [36]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:24:35] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:24:41] Scheduler: Running trials [37]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:24:42] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:24:42] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:24:42] Scheduler: Retrieved COMPLETED trials: [35].
[INFO 02-03 05:24:42] Scheduler: Fetching data for trials: [35].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:24:48] Scheduler: Running trials [38]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:24:49] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:24:49] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:24:49] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:24:50] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:24:51] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:24:53] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:24:57] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:25:02] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:25:10] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 05:25:21] Scheduler: Waiting for completed trials (for 17 sec, currently running trials: 3).
[INFO 02-03 05:25:38] Scheduler: Retrieved COMPLETED trials: 36 - 38.
[INFO 02-03 05:25:38] Scheduler: Fetching data for trials: 36 - 38.
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:25:42] Scheduler: Running trials [39]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:25:42] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:25:47] Scheduler: Running trials [40]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:25:47] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:25:54] Scheduler: Running trials [41]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:25:55] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:25:55] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:25:55] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:25:56] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:25:58] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:26:00] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:26:03] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:26:09] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:26:16] Scheduler: Waiting for completed trials (for 11 sec, currently running trials: 3).
[INFO 02-03 05:26:28] Scheduler: Retrieved COMPLETED trials: [39].
[INFO 02-03 05:26:28] Scheduler: Fetching data for trials: [39].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:26:33] Scheduler: Running trials [42]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:26:34] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:26:34] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:26:34] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:26:35] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:26:37] Scheduler: Retrieved COMPLETED trials: [40].
[INFO 02-03 05:26:37] Scheduler: Fetching data for trials: [40].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:26:44] Scheduler: Running trials [43]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:26:45] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:26:45] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:26:45] Scheduler: Retrieved COMPLETED trials: [41].
[INFO 02-03 05:26:45] Scheduler: Fetching data for trials: [41].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:26:50] Scheduler: Running trials [44]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:26:51] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:26:51] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:26:51] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:26:52] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:26:54] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:26:56] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:27:00] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:27:05] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:27:12] Scheduler: Retrieved COMPLETED trials: [42].
[INFO 02-03 05:27:12] Scheduler: Fetching data for trials: [42].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:27:18] Scheduler: Running trials [45]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:27:19] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:27:19] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:27:19] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:27:20] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:27:21] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:27:23] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:27:27] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:27:32] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:27:40] Scheduler: Retrieved COMPLETED trials: [43].
[INFO 02-03 05:27:40] Scheduler: Fetching data for trials: [43].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:27:45] Scheduler: Running trials [46]...
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:27:46] ax.modelbridge.torch: The observations are identical to the last set of observations used to fit the model. Skipping model fitting.
[INFO 02-03 05:27:46] Scheduler: Generated all trials that can be generated currently. Max parallelism currently reached.
[INFO 02-03 05:27:46] Scheduler: Retrieved COMPLETED trials: [44].
[INFO 02-03 05:27:46] Scheduler: Fetching data for trials: [44].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation./opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:27:52] Scheduler: Running trials [47]...
[INFO 02-03 05:27:53] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 3).
[INFO 02-03 05:27:54] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 3).
[INFO 02-03 05:27:55] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 3).
[INFO 02-03 05:27:58] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 3).
[INFO 02-03 05:28:01] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 3).
[INFO 02-03 05:28:06] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 3).
[INFO 02-03 05:28:14] Scheduler: Retrieved COMPLETED trials: [45].
[INFO 02-03 05:28:14] Scheduler: Fetching data for trials: [45].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:28:14] Scheduler: Done submitting trials, waiting for remaining 2 running trials...
[INFO 02-03 05:28:14] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 2).
[INFO 02-03 05:28:15] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 2).
[INFO 02-03 05:28:16] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 2).
[INFO 02-03 05:28:19] Scheduler: Waiting for completed trials (for 3 sec, currently running trials: 2).
[INFO 02-03 05:28:22] Scheduler: Waiting for completed trials (for 5 sec, currently running trials: 2).
[INFO 02-03 05:28:27] Scheduler: Waiting for completed trials (for 7 sec, currently running trials: 2).
[INFO 02-03 05:28:35] Scheduler: Retrieved COMPLETED trials: [46].
[INFO 02-03 05:28:35] Scheduler: Fetching data for trials: [46].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[INFO 02-03 05:28:35] Scheduler: Waiting for completed trials (for 1 sec, currently running trials: 1).
[INFO 02-03 05:28:36] Scheduler: Waiting for completed trials (for 1.5 sec, currently running trials: 1).
[INFO 02-03 05:28:37] Scheduler: Waiting for completed trials (for 2 sec, currently running trials: 1).
[INFO 02-03 05:28:40] Scheduler: Retrieved COMPLETED trials: [47].
[INFO 02-03 05:28:40] Scheduler: Fetching data for trials: [47].
/opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.OptimizationResult() 评估结果
现在我们可以使用 Ax 提供的辅助函数和可视化工具来检查优化结果。
首先我们生成一个包含实验结果摘要的数据框。该数据框中的每一行对应一个试验即运行的训练作业包含试验的状态、评估的参数配置以及观察到的度量值信息。这提供了一个简单的方法来检查优化的情况。
from ax.service.utils.report_utils import exp_to_dfdf exp_to_df(experiment)
df.head(10) /opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[WARNING 02-03 05:28:40] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column. 试验索引arm 名称试验状态生成方法是否可行参数数量准确率隐藏层大小 1隐藏层大小 2学习率迭代次数丢失率批量大小000_0完成SobolFalse16810.00.90875719660.00318240.19097032111_0完成SobolFalse21926.00.887460231180.00014530.465754256222_0完成SobolTrue37560.00.947588401240.00274540.19660064333_0完成Sobol假14756.00.89309618230.00016640.169496256444_0完成Sobol真71630.00.94892780990.00064220.291277128555_0完成Sobol假13948.00.92269216540.00044420.05755264666_0完成Sobol假24686.00.86377929500.00017720.435030256777_0完成Sobol假18290.00.87703320870.00011940.462744256888_0完成Sobol假20996.00.85943426170.00524510.45581332999_0完成BoTorch真53063.00.962563571250.00197230.17778064
我们还可以可视化验证准确性和模型参数数量之间的权衡的帕累托前沿。
提示
Ax 使用 Plotly 生成交互式图表允许您进行缩放、裁剪或悬停以查看图表组件的详细信息。试试看并查看可视化教程以了解更多信息。
最终优化结果显示在下面的图中其中颜色对应于每次试验的迭代次数。我们看到我们的方法能够成功地探索权衡并找到具有高验证准确性的大型模型以及具有相对较低验证准确性的小型模型。
from ax.service.utils.report_utils import _pareto_frontier_scatter_2d_plotly_pareto_frontier_scatter_2d_plotly(experiment) /opt/conda/envs/py_3.10/lib/python3.10/site-packages/ax/core/map_data.py:188: FutureWarning:The behavior of DataFrame concatenation with empty or all-NA entries is deprecated. In a future version, this will no longer exclude empty or all-NA columns when determining the result dtypes. To retain the old behavior, exclude the relevant entries before the concat operation.[WARNING 02-03 05:28:40] ax.service.utils.report_utils: Column reason missing for all trials. Not appending column. 为了更好地了解我们的代理模型对黑匣子目标学到了什么我们可以看一下留一出交叉验证的结果。由于我们的模型是高斯过程它们不仅提供点预测还提供关于这些预测的不确定性估计。一个好的模型意味着预测的均值图中的点接近 45 度线置信区间覆盖 45 度线并且以期望的频率这里我们使用 95%的置信区间所以我们期望它们在真实观察中包含 95%的时间。
如下图所示模型大小num_params指标比验证准确度val_acc指标更容易建模。
from ax.modelbridge.cross_validation import compute_diagnostics, cross_validate
from ax.plot.diagnostic import interact_cross_validation_plotly
from ax.utils.notebook.plotting import init_notebook_plotting, rendercv cross_validate(modelgs.model) # The surrogate model is stored on the GenerationStrategy
compute_diagnostics(cv)interact_cross_validation_plotly(cv) 我们还可以制作等高线图以更好地了解不同目标如何依赖于两个输入参数。在下图中我们显示模型预测的验证准确度与两个隐藏大小的关系。验证准确度明显随着隐藏大小的增加而增加。
from ax.plot.contour import interact_contour_plotlyinteract_contour_plotly(modelgs.model, metric_nameval_acc) 同样我们在下图中显示模型参数数量与隐藏大小的关系并看到它也随着隐藏大小的增加而增加对hidden_size_1的依赖性更大。
interact_contour_plotly(modelgs.model, metric_namenum_params) 致谢
我们感谢 TorchX 团队特别是 Kiuk Chung 和 Tristan Rice在将 TorchX 与 Ax 集成方面的帮助。
脚本的总运行时间14 分钟 44.258 秒
下载 Python 源代码ax_multiobjective_nas_tutorial.py
下载 Jupyter 笔记本: ax_multiobjective_nas_tutorial.ipynb
Sphinx-Gallery 生成的画廊
