{"id":22225,"date":"2025-04-15T10:49:17","date_gmt":"2025-04-15T03:49:17","guid":{"rendered":"https:\/\/gcloudvn.com\/?p=22225"},"modified":"2025-04-28T08:54:31","modified_gmt":"2025-04-28T01:54:31","slug":"accelerate-ai-ml-workloads-using-cloud-storage-hierarchical-namespace","status":"publish","type":"post","link":"https:\/\/gcloudvn.com\/en\/kienthuc\/accelerate-ai-ml-workloads-using-cloud-storage-hierarchical-namespace\/","title":{"rendered":"Accelerate AI\/ML workloads using Cloud Storage hierarchical namespace"},"content":{"rendered":"<section class=\"wpb-content-wrapper\"><div class=\"vc_row wpb_row vc_row-fluid\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\">\n\t<div class=\"wpb_text_column wpb_content_element\" >\n\t\t<div class=\"wpb_wrapper\">\n\t\t\t<p>As Artificial Intelligence (AI) and Machine Learning (ML) projects become increasingly massive, the supporting infrastructure needs to evolve accordingly to meet the unique needs of businesses. Google Cloud Storage is always striving to provide the AI\/ML community with the most powerful tools, helping to optimize processing performance, flexible scalability according to demand, and superior convenience of cloud storage services. In this article, Gimasys will explore with you how Cloud Storage's new Hierarchical Namespace (HNS) feature can help you fully exploit the performance and efficiency potential of your AI\/ML workloads.<\/p>\n<div id=\"ez-toc-container\" class=\"ez-toc-v2_0_80 counter-hierarchy ez-toc-counter ez-toc-grey ez-toc-container-direction\">\n<div class=\"ez-toc-title-container\">\n<p class=\"ez-toc-title\" style=\"cursor:inherit\">Table of contents<\/p>\n<span class=\"ez-toc-title-toggle\"><a href=\"#\" class=\"ez-toc-pull-right ez-toc-btn ez-toc-btn-xs ez-toc-btn-default ez-toc-toggle\" aria-label=\"Toggle Table of Content\"><span class=\"ez-toc-js-icon-con\"><span class=\"\"><span class=\"eztoc-hide\" style=\"display:none;\">Toggle<\/span><span class=\"ez-toc-icon-toggle-span\"><svg style=\"fill: #999;color:#999\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" class=\"list-377408\" width=\"20px\" height=\"20px\" viewbox=\"0 0 24 24\" fill=\"none\"><path d=\"M6 6H4v2h2V6zm14 0H8v2h12V6zM4 11h2v2H4v-2zm16 0H8v2h12v-2zM4 16h2v2H4v-2zm16 0H8v2h12v-2z\" fill=\"currentColor\"><\/path><\/svg><svg style=\"fill: #999;color:#999\" class=\"arrow-unsorted-368013\" xmlns=\"http:\/\/www.w3.org\/2000\/svg\" width=\"10px\" height=\"10px\" viewbox=\"0 0 24 24\" version=\"1.2\" baseprofile=\"tiny\"><path d=\"M18.2 9.3l-6.2-6.3-6.2 6.3c-.2.2-.3.4-.3.7s.1.5.3.7c.2.2.4.3.7.3h11c.3 0 .5-.1.7-.3.2-.2.3-.5.3-.7s-.1-.5-.3-.7zM5.8 14.7l6.2 6.3 6.2-6.3c.2-.2.3-.5.3-.7s-.1-.5-.3-.7c-.2-.2-.4-.3-.7-.3h-11c-.3 0-.5.1-.7.3-.2.2-.3.5-.3.7s.1.5.3.7z\"\/><\/svg><\/span><\/span><\/span><\/a><\/span><\/div>\n<nav><ul class='ez-toc-list ez-toc-list-level-1' ><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-1\" href=\"https:\/\/gcloudvn.com\/en\/kienthuc\/accelerate-ai-ml-workloads-using-cloud-storage-hierarchical-namespace\/#Vai_tro_cua_luu_tru_trong_khoi_luong_cong_viec_AIML\" >Storage\u2019s role in AI\/ML workloads<\/a><\/li><li class='ez-toc-page-1 ez-toc-heading-level-2'><a class=\"ez-toc-link ez-toc-heading-2\" href=\"https:\/\/gcloudvn.com\/en\/kienthuc\/accelerate-ai-ml-workloads-using-cloud-storage-hierarchical-namespace\/#Loi_ich_cua_viec_su_dung_hierarchical_namespace_cho_khoi_luong_cong_viec_AIML\" >Benefits of using a hierarchical namespace for AI\/ML workloads<\/a><ul class='ez-toc-list-level-3' ><li class='ez-toc-heading-level-3'><a class=\"ez-toc-link ez-toc-heading-3\" href=\"https:\/\/gcloudvn.com\/en\/kienthuc\/accelerate-ai-ml-workloads-using-cloud-storage-hierarchical-namespace\/#Ket_luan\" >Conclusion<\/a><\/li><\/ul><\/li><\/ul><\/nav><\/div>\n<h2><span class=\"ez-toc-section\" id=\"Vai_tro_cua_luu_tru_trong_khoi_luong_cong_viec_AIML\"><\/span>Storage\u2019s role in AI\/ML workloads<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>AI\/ML data pipelines typically consist of the following steps, which can place heavy demands on the underlying storage system:<\/p>\n<ol>\n<li>Data preparation and preprocessing involves data validation, preprocessing, ingesting data into storage and transforming it into the correct format for model training.<\/li>\n<li>Model training is a process which uses many GPU\/TPU compute instances to iteratively develop and refine an AI\/ML model.\nThis process also involves checkpointing, which periodically saves the state of a model so it can be resumed from the last saved state instead of restarting from scratch, saving valuable time and resources. This provides fault tolerance against failures that are common in large-scale distributed training, and also helps developers experiment with hyperparameters or adjust training objectives without losing prior progress.<\/li>\n<\/ol>\n<p><a href=\"https:\/\/gcloudvn.com\/en\/kienthuc\/accelerate-ai-ml-workloads-using-cloud-storage-hierarchical-namespace\/attachment\/thang-72024-2025-04-15t101258-973\/\" rel=\"attachment wp-att-22235\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-22235\" src=\"https:\/\/gcloudvn.com\/wp-content\/uploads\/2025\/04\/Thang-72024-2025-04-15T101258.973.jpg\" alt=\"\" width=\"600\" height=\"375\" srcset=\"https:\/\/gcloudvn.com\/wp-content\/uploads\/2025\/04\/Thang-72024-2025-04-15T101258.973.jpg 600w, https:\/\/gcloudvn.com\/wp-content\/uploads\/2025\/04\/Thang-72024-2025-04-15T101258.973-18x12.jpg 18w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/a><\/p>\n<ol start=\"3\">\n<li>Model serving typically involves loading the model, weights, and dataset into compute instances with GPUs\/TPUs for model inference.\nAI\/ML workloads can run on large compute clusters that consist of thousands of nodes performing simultaneous I\/Os on petabyte-scale datasets. As such, the underlying storage system can often become the bottleneck for AI\/ML pipelines, resulting in underutilization of expensive GPU\/TPU cycles.<\/li>\n<\/ol>\n<h2><span class=\"ez-toc-section\" id=\"Loi_ich_cua_viec_su_dung_hierarchical_namespace_cho_khoi_luong_cong_viec_AIML\"><\/span>Benefits of using a hierarchical namespace for AI\/ML workloads<span class=\"ez-toc-section-end\"><\/span><\/h2>\n<p>Cloud Storage\u2019s hierarchical namespace can be enabled when creating a bucket, and it provides several benefits to AI\/ML workloads, including:<\/p>\n<ul>\n<li aria-level=\"1\">A new \u201cfolder\u201d resource type and APIs that are optimized for filesystem semantics.<\/li>\n<li aria-level=\"1\">Atomic and fast folder renames, resulting in faster and more reliable checkpointing.<\/li>\n<li aria-level=\"1\">An optimized storage layout that handles higher queries per second (QPS) of reads and writes.<\/li>\n<\/ul>\n<p><a href=\"https:\/\/gcloudvn.com\/en\/kienthuc\/accelerate-ai-ml-workloads-using-cloud-storage-hierarchical-namespace\/attachment\/thang-72024-2025-04-15t101324-929\/\" rel=\"attachment wp-att-22234\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-22234\" src=\"https:\/\/gcloudvn.com\/wp-content\/uploads\/2025\/04\/Thang-72024-2025-04-15T101324.929.jpg\" alt=\"\" width=\"600\" height=\"375\" srcset=\"https:\/\/gcloudvn.com\/wp-content\/uploads\/2025\/04\/Thang-72024-2025-04-15T101324.929.jpg 600w, https:\/\/gcloudvn.com\/wp-content\/uploads\/2025\/04\/Thang-72024-2025-04-15T101324.929-18x12.jpg 18w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/a><strong>Data organization and access that\u2019s optimized for filesystem semantics<\/strong><\/p>\n<p>In a hierarchical namespace bucket, folders can contain objects and other folders, which allows organizing (traditionally flat) Cloud Storage data into a tree-like structure that mirrors a traditional filesystem. This lets client libraries like Cloud Storage FUSE map filesystem calls to Cloud Storage APIs that operate directly on folders. While flat namespace buckets often necessitate performing inefficient and costly object-level operations to simulate filesystem operations, using a hierarchical namespace lets you take advantage of filesystem semantics offered natively by the underlying storage system. For example, filesystem libraries typically use resource-intensive ListObject calls to implement inode lookups; these can be replaced with more efficient GetFolderMetadata calls when using a hierarchical namespace. AI\/ML workloads benefit greatly as a result, as they often rely on frameworks like TensorFlow and PyTorch that interact with storage via a filesystem interface.<\/p>\n<p>Customers like AssemblyAI have reported significant improvements using hierarchical namespace with Cloud Storage FUSE to power their AI\/ML workloads.<\/p>\n<p>\u201cWith HNS and GCSfuse we observed over 10x increase in throughput from GCS, with training speed improving 15x.\" <i>\u2013 Ahmed Etefy, Staff Software Engineer, AssemblyAI<\/i><\/p>\n<p><strong>Up to 20x faster checkpointing<\/strong><\/p>\n<p>Renaming folders and objects is common when writing checkpoints or managing intermediate outputs. Cloud Storage\u2019s hierarchical namespace buckets introduce a new RenameFolder API that is both fast and atomic. While simulating a folder rename in a flat namespace bucket could involve thousands of individual object rewrites and deletes (depending on how many objects are in the folder), the hierarchical namespace offering provides a folder-level metadata-only operation that accomplishes this in an atomic action that completes in a fraction of the time. Atomicity prevents inconsistencies and complex state management caused by partial failures, which is a common problem with simulated renames in flat buckets.<\/p>\n<p>Looking at folder renames in action, checkpoint benchmarking shows that hierarchical namespace buckets speed up checkpoint writes by up to 20x compared to flat buckets.<\/p>\n<p><a href=\"https:\/\/gcloudvn.com\/en\/kienthuc\/accelerate-ai-ml-workloads-using-cloud-storage-hierarchical-namespace\/attachment\/thang-72024-2025-04-15t101349-877\/\" rel=\"attachment wp-att-22233\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-full wp-image-22233\" src=\"https:\/\/gcloudvn.com\/wp-content\/uploads\/2025\/04\/Thang-72024-2025-04-15T101349.877.jpg\" alt=\"\" width=\"600\" height=\"375\" srcset=\"https:\/\/gcloudvn.com\/wp-content\/uploads\/2025\/04\/Thang-72024-2025-04-15T101349.877.jpg 600w, https:\/\/gcloudvn.com\/wp-content\/uploads\/2025\/04\/Thang-72024-2025-04-15T101349.877-18x12.jpg 18w\" sizes=\"auto, (max-width: 600px) 100vw, 600px\" \/><\/a><\/p>\n<p><strong>Up to 8x higher QPS<\/strong><\/p>\n<p>AI\/ML workloads running on large clusters generate millions of I\/O requests on the attached storage system. Checkpoint writes and restores during model training and serving reads for inference are highly bursty workloads where many nodes are synchronized to talk to storage at the same time. High QPS capabilities help avoid storage bottlenecks that could starve expensive GPUs\/TPUs.<\/p>\n<p>Hierarchical namespace buckets have an optimized storage layout that provides up to 8x higher initial object read and write requests per second (QPS) compared to flat namespace buckets, while still supporting a doubling of the QPS every 20 minutes per the <a href=\"https:\/\/cloud.google.com\/storage\/docs\/request-rate#ramp-up\" target=\"_blank\" rel=\"noopener\">h\u01b0\u1edbng d\u1eabn t\u0103ng t\u1ed1c c\u1ee7a Cloud Storage<\/a> For example, this means a cold hierarchical namespace bucket can achieve 100,000 object write QPS in nearly half the time compared to a flat bucket.<\/p>\n<h3><span class=\"ez-toc-section\" id=\"Ket_luan\"><\/span>Conclusion<span class=\"ez-toc-section-end\"><\/span><\/h3>\n<p>AI\/ML workloads require infrastructure tailored to their unique needs: efficient data organization and file system semantics for tight integration with frameworks, high performance benchmarking to maximize GPU\/TPU utilization, and high QPS ratios to support rapid acceleration. Hierarchical namespaces provide all of these benefits, along with the scalability, reliability, simplicity, and cost-effectiveness that Cloud Storage is known for. Gimasys recommends enabling hierarchical namespaces on new buckets for AI\/ML workloads.<\/p>\n\n\t\t<\/div>\n\t<\/div>\n<div class=\"vc_empty_space\"   style=\"height: 32px\"><span class=\"vc_empty_space_inner\"><\/span><\/div><div class=\"templatera_shortcode\"><div class=\"vc_row wpb_row vc_row-fluid\"><div class=\"wpb_column vc_column_container vc_col-sm-12\"><div class=\"vc_column-inner\"><div class=\"wpb_wrapper\"><div class=\"vc_message_box vc_message_box-standard vc_message_box-rounded vc_color-blue\" ><div class=\"vc_message_box-icon\"><i class=\"vc-mono vc-mono-technorati\"><\/i><\/div><p><a href=\"https:\/\/gcloudvn.com\/en\/main-logo-1\/\"><img loading=\"lazy\" decoding=\"async\" class=\"alignleft wp-image-664\" src=\"https:\/\/gcloudvn.com\/wp-content\/uploads\/2021\/06\/main-logo-1.png\" alt=\"\" width=\"221\" height=\"72\" srcset=\"https:\/\/gcloudvn.com\/wp-content\/uploads\/2021\/06\/main-logo-1.png 214w, https:\/\/gcloudvn.com\/wp-content\/uploads\/2021\/06\/main-logo-1-18x6.png 18w, https:\/\/gcloudvn.com\/wp-content\/uploads\/2021\/06\/main-logo-1-183x60.png 183w\" sizes=\"auto, (max-width: 221px) 100vw, 221px\" \/><\/a>As a senior partner of Google in Vietnam, Gimasys has more than 10+ years of experience, consulting on implementing digital transformation for 2000+ domestic corporations. Some typical customers Jetstar, Dien Quan Media, Heineken, Jollibee, Vietnam Airline, HSC, SSI...<\/p>\n<p>Gimasys is currently a strategic partner of many major technology companies in the world such as Salesforce, Oracle Netsuite, Tableau, Mulesoft.<\/p>\n<p>Contact Gimasys - Google Cloud Premier Partner for advice on strategic solutions suitable to the specific needs of your business:<\/p>\n<ul>\n<li>Email: gcp@gimasys.com<\/li>\n<li>Hotline: 0974 417 099<\/li>\n<\/ul>\n<\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/div><\/div>\n<\/section>","protected":false},"excerpt":{"rendered":"Khi c\u00e1c d\u1ef1 \u00e1n Tr\u00ed tu\u1ec7 Nh\u00e2n t\u1ea1o (AI) v\u00e0 H\u1ecdc m\u00e1y (ML) ng\u00e0y c\u00e0ng \u0111\u1ed3 s\u1ed9, h\u1ea1 t\u1ea7ng h\u1ed7 tr\u1ee3 c\u1ea7n ph\u1ea3i ph\u00e1t tri\u1ec3n t\u01b0\u01a1ng \u1ee9ng \u0111\u1ec3 \u0111\u00e1p \u1ee9ng nh\u1eefng nhu c\u1ea7u ri\u00eang bi\u1ec7t c\u1ee7a doanh nghi\u1ec7p. Google Cloud&hellip;","protected":false},"author":2,"featured_media":22231,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"inline_featured_image":false,"footnotes":""},"categories":[1,135],"tags":[],"class_list":["post-22225","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-kienthuc","category-google-cloud-platform","entry","has-media"],"_links":{"self":[{"href":"https:\/\/gcloudvn.com\/en\/wp-json\/wp\/v2\/posts\/22225","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/gcloudvn.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/gcloudvn.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/gcloudvn.com\/en\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/gcloudvn.com\/en\/wp-json\/wp\/v2\/comments?post=22225"}],"version-history":[{"count":0,"href":"https:\/\/gcloudvn.com\/en\/wp-json\/wp\/v2\/posts\/22225\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/gcloudvn.com\/en\/wp-json\/wp\/v2\/media\/22231"}],"wp:attachment":[{"href":"https:\/\/gcloudvn.com\/en\/wp-json\/wp\/v2\/media?parent=22225"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/gcloudvn.com\/en\/wp-json\/wp\/v2\/categories?post=22225"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/gcloudvn.com\/en\/wp-json\/wp\/v2\/tags?post=22225"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}