5 ways to get your data into Cloud Storage

14/04/2021

Once you have created it buckets and now you want to use the power of the cloud to deliver your content. With the to-do attitude and insights of this post, you will learn how to get data into Cloud Storage using a variety of data upload methods.

When you upload an object to the group Storage Cloud Your object will include the data you want to store, along with any associated metadata. When it comes to the actual upload, you have a few different options to choose from which we'll look at below. For more details, please Document Viewer. And for general, conceptual information about uploading and downloading, Please read this.

Table of contents

Cloud Console – Cloud Console

The first thing to mention is the Cloud Console. This gives you an in-browser experience where you can easily click to create groups and folders, then select or drag and drop files from your local machine to upload.

Pro tip: folder uploads are also supported if you're using the Chrome browser.

gsutil tool

For production environments, you may want an automated, command-line solution.

For this, Google provides the gsutil tool. gsutil is a Python application that allows you to access Cloud Storage from the command line, giving you the ability to do all sorts of things like create buckets, move objects, or even edit metadata.

To use it, run the gsutil program with many command line options. For example, this command uploads a folder of files from your local machine to your Cloud Storage bucket using parallel upload.

gsutil tool

And this command lists specific objects with instance-specific URLs using wildcards.

gsutil tool 2

You can find more interesting things you can do with the gsutil tool in this document.

Client libraries

At some point, you may need to interface with Cloud Storage directly from your code, rather than going outside a command line option. You can include client libraries in your code and call a simple api to get data into a bucket or folder.

And before you ask about languages, with options in C++, C #, Go, Java, Node.js, PHP, Python and Ruby — we've got you covered.

Eg:

Take a look at this Python code to upload an object to a Cloud Storage bucket:

Language: Python

from google.cloud import storage def upload_blob(bucket_name, source_file_name, destination_blob_name): """Uploads a file to the bucket.""" # bucket_name = "your-bucket-name" # source_file_name = "local/path/to/file " # destination_blob_name = "storage-object-name" storage_client = storage.Client() bucket = storage_client.bucket(bucket_name) blob = bucket.blob(destination_blob_name) blob.upload_from_filename(source_file_name)

Check out more designs here.

JSON and XML

And finally, if none of that does the trick, there's always the JSON and XML APIs, which can let you initiate an HTTP POST request to upload data directly to a bucket or directory. item. It's a bit more complicated, but there if you need it.

POST /OBJECT_NAME HTTP/2 Host: BUCKET_NAME.storage.googleapis.com Date: DATE Content-Length: REQUEST_BODY_LENGTH Content-Type: MIME_TYPE X-Goog-Resumable: start Authorization: AUTHENTICATION_STRING

Cloud Storage Transfer Appliance

Now, for those of you with a LOT of data, it's worth noting that uploading it all directly from your on-premises system Google Cloud Platform may not be possible – so you can use the Cloud Storage Transfer Tool.

> Reference:

Google presents you with a device, you connect it, add your data and send it back to google. Plus, you can leave this fun-looking box on your desk for a while, which can be a great conversation starter, if you're into that sort of thing. More details here.

More clouds, more problems? Not so!

Don't worry if your data is in another cloud, Google has it instruct easy to use to get you up and running with supporting multi-cloud environments and moving that data to Cloud Storage.

So what now?

Of course, now that the data is in Cloud Storage, you have to figure out the best ways to distribute that data to your users around the world. Stay tuned for best practices for getting that data out into the world in the next Google Cloud post.

Learn more about your storage options in Cloud Storage Bytes (Link: https://youtube.com/playlist?list=PLIivdWyY5sqJcBvDh5dfPoblLGhG1R1-O) or view document for more information, including instructions.

Contact Gimasys for advice on transformation strategies that are suitable for your business situation and free Google Cloud service experience: