Researchers from SONY today announced a new speed record for training ImageNet/ResNet 50 in only 224 seconds (three minutes and 44 seconds) with 75 percent accuracy using 2,100 NVIDIA Tesla V100 Tensor Core GPUs. This achievement represents the fastest reported training time ever published on ResNet-50.
The team also achieved over 90% GPU scaling efficiency with 1,088 NVIDIA Tesla V100 Tensor Core GPUs.
GPU scaling efficiency with ImageNet/ResNet-50 training
| Processor | Interconnect | GPU scaling efficiency | |
| Goyal et al. [1] | Tesla P100 x256 | 50Gbit Ethernet | ∼90% |
| Akiba et al. [5] | Tesla P100 x1024 | Infiniband FDR | 80% |
| Jia et al. [6] | Tesla P40 x2048 | 100Gbit Ethernet | 87.90% |
| This work | Tesla V100 x1088 | Infiniband EDR x2 | 91.62% |
Training time and top-1 validation accuracy with ImageNet/ResNet-50“As the size of datasets and deep neural network (DNN) model for deep learning increase, the time required to train a model is also increasing,” the SONY team wrote in their paper.
| Batch Size | Processor | DL Library | Time | Accuracy | |
| He et al. | 256 | Tesla P100 x8 | Caffe | 29 hours | 75.30% |
| Goyal et al. | 8K | Tesla P100 x256 | Caffe2 | 1 hour | 76.30% |
| Smith et al. | 8K→16K | full TPU Pod | TensorFlow | 30 mins | 76.10% |
| Akiba et al. | 32K | Tesla P100 x1024 | Chainer | 15 mins | 74.90% |
| Jia et al. | 64K | Tesla P40 x2048 | TensorFlow | 6.6 mins | 75.80% |
| This work | 34K→68K | Tesla V100 x2176 | NNL | 224 secs | 75.03% |
\To achieve the record, the researchers addressed two primary issues with large-scale distributed training: instability of large mini-batch training and the synchronization communication overhead, the team said.
“We adopt a batch size control technique to address large mini-batch instability,” the researchers said. “We [also] develop a 2D-Torus all-reducing scheme to efficiently exchange gradients across GPUs.”
The 2D-Torus serves as an efficient communication topology that reduces the communications overhead of a collective operation.
Software: “We used Neural Network Libraries (NNL) and its CUDA extension as a DNN training framework,” the team said. “We also used development branches based on NNL version 1.0.0. CUDA version 9.0 with cuDNN version 7.3.1 is employed to train DNN in GPUs.”
“We used NCCL version 2.3.5 and OpenMPI version 2.1.3 as communication libraries. The 2D-Torus all-reduce is implemented with NCCL2. The above software is packaged in a Singularity container. We used Singularity version 2.5.2 to run distributed DNN training.,” the team wrote in their paper.
The work leveraged the development framework “Core Library: Neural Network Libraries developed by Sony, and the AI Bridging Cloud Infrastructure (ABCI) supercomputer, a world-class computing infrastructure constructed and operated by Japan’s National Institute of Advanced Industrial Science and Technology (AIST). The system is powered by over 4,300 NVIDIA Volta Tensor Core GPUs.
Read more>
