{"id":486,"date":"2025-11-25T09:25:02","date_gmt":"2025-11-25T01:25:02","guid":{"rendered":"https:\/\/high-flyer.in.suopu.cc\/?p=486"},"modified":"2025-11-25T09:25:02","modified_gmt":"2025-11-25T01:25:02","slug":"3fs%e4%bc%98%e5%8c%96-02-%e5%ae%a2%e6%88%b7%e7%ab%af%e5%86%85%e5%ad%98%e4%bd%bf%e7%94%a8%e4%bc%98%e5%8c%96","status":"publish","type":"post","link":"https:\/\/high-flyer.in.suopu.cc\/en\/blog\/486\/","title":{"rendered":"3FS Optimization 02 | Client Memory Usage Optimization"},"content":{"rendered":"

e.g. articlePhantom Force | High Speed File Series 3FS<\/a>Phantom AI has designed a sample read file system, 3FS, that is ideal for deep learning training. 3FS uses Direct IO and RDMA Read to allow model training to achieve high read bandwidth with minimal CPU and memory overhead in the sample read section, eliminating the need to wait for data to be loaded during the training process, and utilizing the GPU more fully. performance of GPUs.<\/p>\n

As we know, file systems are generally categorized into client and server side. In a 3FS file system, the client is deployed on the compute node side and accesses the 3FS server deployed on the storage node side over the network. There are many issues in this process that are worth optimizing. In this issue, we will talk to you about the client side of theMemory usage optimization<\/strong>of the problem.<\/p>\n

Phantom AI's deep learning platform is separate from computation and storage, and the computation cluster is only responsible for computation. Whether it's data reading, model loading, or gradient reduce, etc., all of them need to be transferred to memory and then to video memory through the NIC, and in the middle of the process, it needs to be copied many times, which is a very serious use of the memory bandwidth. If we can effectively optimize the use of memory, the performance of model training will be greatly improved.<\/p>\n

summarize<\/h2>\n

The client side of a 3FS is the 3FS deployed on the compute node side, whose role is to connect with the 3FS on the server side over the network, request data from the server side and parse it. For the Phantom Firefly II cluster, there is a high-speed network card on each machine in the cluster, and for a typical file system, let's look at the memory overhead required to read 23.5GB of data per second:<\/p>\n

    \n
  1. Read from NIC to kernel-state memory: one memory write<\/li>\n
  2. Read from kernel state to Page Cache, one memory read, one memory write<\/li>\n
  3. Reading from Page Cache to userland memory: one memory read and one memory write<\/li>\n
  4. In addition, since the register instructions for Non-temporal store are not supported inside the kernel, one to two times more memory reads are required for writes. For more information, seehere are<\/a>\u3002<\/li>\n<\/ol>\n

    As mentioned above, if we want to achieve a read performance of 23.5 GBps, we need to consume per second\u00a023.5 * 6~7 = 141~165GBps<\/code>\u00a0of memory bandwidth.<\/p>\n

    And the most advanced training platforms on the market today, such as the dual-path AMD Epyc Rome\/Milan have a total memory bandwidth in the range of270GBps~330GBps<\/code>The total memory bandwidth of a dual-lane Intel Cascade Lake platform is only about 220GBps, which means that for a typical file system, the memory bandwidth consumed just to read data will take up about50%~70%<\/code>! And while we are using these machines, it is clear that only data is not enough, and if IO takes up so much bandwidth, there are fewer resources left for computation and communication.<\/p>\n

    Therefore, reducing memory copy operations and lowering memory overhead are the key directions for us to optimize the performance of 3FS clients. We useRDMA Read<\/strong>of technology to solve that type of problem.<\/p>\n

    RDMA Read<\/h2>\n

    weRead to userland memory via RDMA<\/strong>To further avoid kernel-to-user memory copying. As in the previous example, after using RDMA Read, the read operation becomes one step: read from the NIC to the user memory directly, i.e., reading a full 23.5GBps NIC consumes only 23.5GBps of memory bandwidth, which is less than 10% of the total memory bandwidth.<\/p>\n

    We chose to initiate an RDMA Read from the compute node side instead of doing an RDMA Write from the storage node side.This is because the bulk reads of 3FS result in read requests on the same network that may correspond to many application Buffers, and RDMA Write does not support distributing to multiple addresses on the receiver side, which means that when the length of a single read is very small, it will generate many very small RDMA requests, to the point of significantly affecting the bandwidth of the read.<\/p>\n

    \"image.png\"<\/a><\/span><\/p>\n

    However, it is important to note that the read data has to be aligned, and RDMA Read does not simply fill the user's buffer with the read data; instead, it needs to read the data used for alignment on the storage node side into the padding buffer allocated in the kernel, and then discard the data after alignment on the compute node side. During performance testing, we found that RDMA Read requests of different sizes affect each other:<\/p>\n