单页面企业网站,江苏省建设教育协会网站,中文字幕mv在线观看,企业建站系统费用今天跑huggingface的示例的时候#xff0c;遇到了最让我头疼的问题#xff0c;国内网上还没有对应的解释#xff0c;我可能是第一人#xff08;汗#xff09;先看看报错#xff1a;
Traceback (most recent call last):File F:\transformer\transformers\examples…今天跑huggingface的示例的时候遇到了最让我头疼的问题国内网上还没有对应的解释我可能是第一人汗先看看报错
Traceback (most recent call last):File F:\transformer\transformers\examples\pytorch\image-classification\run_image_classification.py, line 451, in modulemain()File F:\transformer\transformers\examples\pytorch\image-classification\run_image_classification.py, line 425, in maintrain_result trainer.train(resume_from_checkpointcheckpoint)File F:\anaconda\envs\gpu\lib\site-packages\transformers-4.38.0.dev0-py3.9.egg\transformers\trainer.py, line 1597, in trainreturn inner_training_loop(File F:\anaconda\envs\gpu\lib\site-packages\transformers-4.38.0.dev0-py3.9.egg\transformers\trainer.py, line 1635, in _inner_training_looptrain_dataloader self.get_train_dataloader()File F:\anaconda\envs\gpu\lib\site-packages\transformers-4.38.0.dev0-py3.9.egg\transformers\trainer.py, line 845, in get_train_dataloaderreturn self.accelerator.prepare(DataLoader(train_dataset, **dataloader_params))File F:\anaconda\envs\gpu\lib\site-packages\torch\utils\data\dataloader.py, line 241, in __init__raise ValueError(prefetch_factor option could only be specified in multiprocessing.
ValueError: prefetch_factor option could only be specified in multiprocessing.let num_workers 0 to enable multiprocessing.可以看到报错信息是prefetch_factor这个属性只有在num_workers大于0的时候才能被定义。但是我去看了看源码很明显train源码中定义没有问题啊 然后我在github上的pytorch的issue中发现了有人提出了疑问问题在此 if num_workers 0 and prefetch_factor ! 2: raise ValueError(prefetch_factor option could only be specified in multiprocessing. let num_workers 0 to enable multiprocessing.) 上面这段是pytorch的源码num_workers表示要用于数据加载的子进程数prefetch_factor表示提前加载的batch数也就是每个 worker 提前加载 的 sample 数量。虽然我现在不是很懂为什么要这么写但是我可以确定的是当num_workers0的时候prefetch_factor应该是没有意义的关于里面的细节我没有深入希望明白原理的人看到之后能在评论区补充谢谢所以源码这么写就一定会报错了然后我根据github上某人的代码修改如下
if num_workers 0:if prefetch_factor is None:prefetch_factor 2 # default value
else:if prefetch_factor is not None:raise ValueError(prefetch_factor option could only be specified in multiprocessing. let num_workers 0 to enable multiprocessing, otherwise set prefetch_factor to None.)成功运行起来了这是跑示例遇到的最后一个坑感谢
