Huawei has recently launched new AI storage products specifically designed for the large-scale model era. These products offer optimal storage solutions for various types of model training, including basic model training, industry model training, and segmented model training and reasoning. With these advancements, Huawei aims to unleash new momentum for AI.
When it comes to developing and implementing large-scale model applications, enterprises face several significant challenges. Firstly, data preparation can be time-consuming due to long processing times, scattered data sources, and slow data collection. Preprocessing 100 TB of data alone can take up to 10 days. Secondly, the utilization of massive text and image training sets in multi-modal large models results in low loading efficiency, with current loading speeds for small files being less than 100 MB/s. This hampers the overall efficiency of the training process.
Third, large models often require frequent parameter adjustments, but unstable training platforms cause interruptions, with an average of one interruption every two days. To resume training, a checkpoint mechanism is necessary, which further prolongs the recovery time. Lastly, implementing large models comes with high thresholds, complicated system construction, resource scheduling challenges, and low GPU resource utilization rates, often below 40%.
To align with the AI development trends of the large-scale model era, Huawei has introduced two innovative products: the OceanStor A310 deep learning data lake storage and the FusionCube A3000 training/super-integration all-in-one machine. These solutions cater to various industries and scenarios, providing comprehensive support for large-scale model applications.
Huawei’s new AI storage products aim to tackle the challenges faced by enterprises in the era of large-scale models. By addressing issues related to data preparation, loading efficiency, training interruptions, and implementation thresholds, Huawei empowers businesses to leverage the full potential of AI in this exciting new era.
Zhou Yuefeng, president of Huawei’s data storage product line, said: “In the era of large-scale models, data determines the height of AI intelligence. As the carrier of data, data storage has become a key infrastructure for AI large-scale models. Huawei’s data storage will continue to innovate in the future, facing the era of AI large-scale models Provide a variety of solutions and products, and work with partners to promote AI empowerment in various industries.”
Join Us On Telegram and Google News
