GCP Classroom Series – 31/Mar/2021

Google Cloud Storage (GCS)

  • GCS is used to store the unstructured data such as images, vidoes and other static content as well as backups, disaster recovery.

  • GCS stores data in the form of objects on an underlying proprietary distributed file system called as Colossus

  • We can transfer the data in our out of GCS using gsutil

  • Different offerings by Google based on usecase Preview

  • Cloud Storage is simple an user friendly (think of Google Drive)

  • We dont need to allocate capacity

  • We can store files till infinity. GCS supports datasets of any size

  • Each file cannot have size more than 5 TB

  • This supports 5000 writes adn 1000 reads per second

  • In GCS when we want to store the data we need to create the storage buckets

  • Basic Terminology

    • Buckets are basic containers that hold your data. Each Bucket has collection of objects
    • Each bucket has a name which must be globally unique
    • Objects are individual pieces of data that you store in Cloud Storage
    • Object names are treated as object metadata. Object names should be less that 1024 bytes in length.
    • Object name should be unique with in bucket
    • Example object names
      • myphoto.jpeg
      • /year2021/jan/myphoto.jpeg
  • The access to the data in the buckets can be done as shown below Preview

  • The objects in the GCS bucket are categories into two sections

    • Cold data: Accessed infrequently
    • Hot Data: Accessed frequently

Storage Classes

  • Buckets are mainly divided into four classes based on their availabilty,demographic relevance and frequency of access Preview
  • Hot and available – Multi Regional:
    • This is Geo-Redundant. When we upload any data in this storage class two separte copies will be stored in two regions of gcp which are hundreds of miles away from each other
    • This is most expensive class of buckets and should be used only when we are certain of global traffic.
  • Hot and local – Regional:
    • This is used for webapplications with highly concentrated traffic patterns with in a region
  • Cool-Nearline: We use this storage class when we expect access data less frequently (once in a month). Here we pay less for storage and more for access
  • Coldline: This is a cold storage facility for data that we would expect to access less than once in a year. Suitable for disaster recovery & long-term archival storage.
  • Lets estimate some costs use pricing calculator Refer Here
    • We need to store 10 TB of data Preview Preview
    • Multi-regional Preview

Working with GCS buckets

  • Creating Buckets:
    • For creating buckets in GCP there are multiple ways Console, gsutil, REST API.
    • To create bucket we need three fiels
      • Region
      • Storage Class
      • Univesally unique name
  • Creating a bucket using Web Console Preview Preview Preview Preview Preview Preview Preview Preview Preview
  • Creating buckets using gsutil:
    • For overview Refer Here and for installing if you dont have gsutil installed Refer Here
    • For command reference use the left hand pane Refer Here Preview
    • Lets create a bucket using gsutil Preview
    • Refer Here for the cli created in the class

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

About learningthoughtsadmin