{"id":501,"date":"2025-11-25T09:28:18","date_gmt":"2025-11-25T01:28:18","guid":{"rendered":"https:\/\/high-flyer.in.suopu.cc\/?p=501"},"modified":"2025-11-25T09:28:18","modified_gmt":"2025-11-25T01:28:18","slug":"pytorch-%e5%88%86%e5%b8%83%e5%bc%8f%e8%ae%ad%e7%bb%83%e6%96%b9%e6%b3%95","status":"publish","type":"post","link":"https:\/\/high-flyer.in.suopu.cc\/en\/blog\/501\/","title":{"rendered":"PyTorch Distributed Training Method"},"content":{"rendered":"

\"image.png\"<\/a><\/span><\/p>\n

In 2018, the Bert model with nearly 300 million parameters came out of nowhere, pushing the NLP field to new heights. In recent years, the development of the artificial intelligence field has increasingly tended to the study of large models, and all major AI giants have released their large models with hundreds of billions of parameters, giving birth to many new AI application scenarios. On the other hand, a variety of factors continue to promote the significant development of big models: 1) society is experiencing a deep digital transformation, and a large amount of data is gradually merging, giving rise to many AI application scenarios and needs; 2) hardware technology continues to progress: NVIDIA A100 GPU, Google's TPU, Ali's Contained Light 800, etc., driving the AI computing power and so on, pushing the AI arithmetic power to be continuously improved. It can be said that the combination of \u201cbig data + big model + big computing power\u201d is the cornerstone of AI 2.0. However, for most AI practitioners, due to a number of factors, it is usually difficult to come into contact with big data or big computing power, and many graduate students in colleges and universities tend to have a lot of experience in AI.Accustomed to using one graphics card (personal PC) to accelerate training<\/strong>This not only slows down training, but also restricts imagination and imprisons creativity. Mirage AI has built \u201cFirefly II\u201d according to its own business needs, aiming to provide a surging deep learning computing power service like the sea. The team self-researched intelligent time-sharing scheduling system, efficient storage system and network communication system, the cluster can be used as an ordinary computer, according to the task demand elastic expansion of GPU arithmetic; self-research hfai data warehouse, model warehouse, optimization of the AI framework and arithmetic, integration of many cutting-edge application scenarios; Client interfaces or Jupyter can be easily accessed to accelerate the training of AI models. The AI model can be easily accessed through the Client interface or Jupyter, accelerating the training of the AI model.<\/p>\n

This installment of the article shares, theHow to use up multiple graphics cards to speed up your AI models<\/strong>Distributed training techniques are becoming one of the essential skills for AI practitioners. Distributed training techniques are becoming one of the essential skills for AI practitioners, which is the way from \u201csmall model\u201d to \u201cbig model\u201d. Let's take ResNet training written in PyTorch as an example to show you different distributed training methods and their effects.<\/p>\n

training task<\/strong>: ResNet + ImageNet<\/p>\n

Training framework<\/strong>: PyTorch 1.8.0<\/p>\n

Training platforms<\/strong>: Phantom Firefly II<\/p>\n

Training code<\/strong>\uff1ahttps:\/\/github.com\/HFAiLab\/pytorch_distributed<\/a><\/p>\n

Training preparation<\/h3>\n

Firefly II currently fully supports the parallel training environment of PyTorch. I first use 1 node with 8 cards to test the time required for different distributed training methods, graphics card utilization efficiency; after that, multiple nodes are used to verify the effect of parallel acceleration. The method of testing is as follows:<\/p>\n

    \n
  1. nn.DataParallel<\/li>\n
  2. torch.distributed + torch.multiprocessing<\/li>\n
  3. apex Half precision<\/li>\n<\/ol>\n

    To fully utilize the video memory, herebatch_size<\/code>Set it to 400 and record the elapsed time per Epoch for comparison. We started with a single-computer, single-card test as the BASELINE, which took 1786 seconds up and down per Epoch.<\/p>\n

    The test results found thatParallel acceleration with half-precision adds the best results<\/strong>\uff1bDataParallel is slower and does not utilize video card resources as well, not recommended!<\/strong>\uff1bThe acceleration from multiple machines and cards is significant<\/strong>. The overall results are as follows:<\/p>\n

    \"result.png\"<\/a><\/span><\/p>\n

    nn.DataParallel<\/h3>\n

    DataParallel is an early data parallel training method proposed by PyTorch, which uses single-process control to load models and data into multiple GPUs, control the flow of data between GPUs, and collaborate with models on different GPUs for parallel training.<\/p>\n

    It's very easy to use, we just need to use the\u00a0nn.DataParallel<\/code>\u00a0Packaging model, and then set some parameters can be set. The parameters to be defined include:<\/p>\n