{"id":477,"date":"2025-11-25T09:22:50","date_gmt":"2025-11-25T01:22:50","guid":{"rendered":"https:\/\/high-flyer.in.suopu.cc\/?p=477"},"modified":"2025-11-25T09:22:50","modified_gmt":"2025-11-25T01:22:50","slug":"hfai-python-%e4%bb%bb%e5%8a%a1%e6%8f%90%e4%ba%a4%e4%bb%bb%e6%84%8f%e6%89%80%e8%87%b3%ef%bc%8c%e8%90%a4%e7%81%ab%e8%ae%ad%e7%bb%83%e8%a1%8c%e4%ba%91%e6%b5%81%e6%b0%b4","status":"publish","type":"post","link":"https:\/\/high-flyer.in.suopu.cc\/en\/blog\/477\/","title":{"rendered":"hfai python | Task submission at will, Firefly training on the fly"},"content":{"rendered":"

Phantom AI has released its years-long sedimentaryDeep Learning Suite hfai<\/a>\u00a0It has attracted many peer researchers and developers to inquire about the trial. The whole suite has more functions, and after familiarizing with this set of rules, it is possible to easily call up the platform's arithmetic resources, so as to efficiently complete the training task.<\/p>\n

To this end, we have created the \u201chfai use of the heart\u201d series of albums, episodes one after another to introduce you to the hfai some of the functions of the design ideas and principles, to help you better and faster to learn the heart, with hfai this set of \u201cmagic skills\u201d easily The hfai is a set of \"divine powers\" that can be used to easily deal with the challenges of deep learning assignments, so that the weight of the work can be lifted easily, and the examples can be used.<\/p>\n

The last two moves introduced you to the\u00a0hfai workspace<\/a>\u00a0\u548c\u00a0haienv<\/a>It can help users to quicklySynchronize local (PC, personal cluster, etc.) project directory (code, data), environment<\/strong>To remote clusters. This set of combinations down, in fact, we can think of it as a \u201cbuild up\u201d and \u201cboost\u201d process, the next is the most central part of the \u201cdivine power\u201d, this article will introduce you to\u00a0hfai python<\/code>It can help everyoneEasily and quickly initiate and manage training tasks<\/strong>\u3002<\/p>\n

Usage Scenarios<\/h2>\n

\u5982As described in previous articles<\/a>After users transfer local data, code, and environment to the remote firefly cluster, they can then submit tasks to train the model. However, how to Firefly cluster computing power evenly and reasonably allocated to different tasks, and how users can view the management of their own tasks?<\/p>\n

All of these functions can be accessed through the\u00a0hfai python<\/code>\u00a0to realize.<\/p>\n

basic concept<\/h2>\n

Some basic concepts are introduced before they are used.<\/p>\n

Phantom AI presents theTask-level time-sharing scheduling<\/strong>the concept of managing firefly clusters.Arithmetic power is allocated in terms of tasks rather than users<\/strong>. Users are free to submit tasks and apply for arithmetic scale freely, and the resources are uniformly allocated by the cluster scheduling system. As shown in the figure below:<\/p>\n

\"01.png\"<\/a><\/span><\/p>\n

Computing resources are sliced and diced in the time dimension, and computing power is allocated to different tasks. This approach not only improves the overall utilization rate of the cluster, but also facilitates users to flexibly scale the card for different tasks, so that everyone has the opportunity to call on large-scale computing power for AI research.<\/p>\n

\"04.png\"<\/a><\/span><\/p>\n

On the Firefly cluster, training tasks are prioritized, with higher priority tasks able to preempt lower priority arithmetic resources. Tasks within the same priority level are organized as \u201cfirst come first served<\/strong>\u201dThe principle of providing arithmetic resources. hfai provides a number of convenient tools (for example:hfai.client<\/a>\uff0chfai.checkpoint<\/a>) to facilitate your code during the training processReceive cluster scheduling information, set breakpoints for saving<\/strong>\u3002<\/p>\n

Local Code Debugging<\/h2>\n

In general, we normally perform deep learning training, such as training the\u00a0Bert<\/a>\uff1a<\/p>\n

\n
python bert.py -c large.yml<\/code><\/pre>\n<\/div>\n
When you need to train a model using phantom firefly, you can debug locally to fix the problem before submitting it to a cluster run. hfai provides signals to simulate interruptions to help you test the suitability of the cluster run method, as in the following example:<\/p>\n
\n
hfai python bert.py -c large.yml ++ --suspend_seconds 100<\/span> --life_state 1<\/span><\/code><\/pre>\n<\/div>\n
here are<\/p>\n