Ingesting Data Into The Cloud
In this lab you'll learn how to use a bash script to download selected data, which provides historic information about internal flights in the United States, from a large public data set available on the internet.
The techniques used to ingest this data from the US Bureau of Transport Statistics website into the cloud can be applied generally to other data sets that provide comprehensive real world data that needs to be parsed and cleaned up before it can be used.
This data will be used in other labs in the Data Science on Google Cloud Platform quest to demonstrate a wide range of data science concepts and techniques using the Google Cloud Platform.
Retrieve initial data from the BTS website.
Store the data on Google Cloud Storage
Setup and Requirements
Before you click the Start Lab button
Read these instructions. Labs are timed and you cannot pause them. The timer, which starts when you click Start Lab, shows how long Cloud resources will be made available to you.
This Qwiklabs hand-on lab lets you do the lab activities yourself in a real cloud environment, not in a simulation or demo environment. It does so by giving you new, temporary credentials that you use to sign in and access the Google Cloud Platform for the duration of the lab.
What you need
To complete this lab, you need:
- Access to a standard internet browser (Chrome browser recommended).
- Time to complete the lab.
Note: If you already have your own personal GCP account or project, do not use it for this lab.
How to start your lab and sign in to the Console
Click the Start Lab button. If you need to pay for the lab, a pop-up opens for you to select your payment method. On the left you will see a panel populated with the temporary credentials that you must use for this lab.
Copy the username, and then click Open Google Console. The lab spins up resources, and then opens another tab that shows the Choose an account page.
Tip: Open the tabs in separate windows, side-by-side.
On the Choose an account page, click Use Another Account.
The Sign in page opens. Paste the username that you copied from the Connection Details panel. Then copy and paste the password.
Important: You must use the credentials from the Connection Details panel. Do not use your Qwiklabs credentials. If you have your own GCP account, do not use it for this lab (avoids incurring charges).
Click through the subsequent pages:
- Accept the terms and conditions.
- Do not add recovery options or two-factor authentication (because this is a temporary account).
- Do not sign up for free trials.
After a few moments, the GCP console opens in this tab.
Activate Google Cloud Shell
Google Cloud Shell is a virtual machine that is loaded with development tools. It offers a persistent 5GB home directory and runs on the Google Cloud. Google Cloud Shell provides command-line access to your GCP resources.
In GCP console, on the top right toolbar, click the Open Cloud Shell button.
In the dialog box that opens, click START CLOUD SHELL:
It takes a few moments to provision and connect to the environment. When you are connected, you are already authenticated, and the project is set to your PROJECT_ID. For example:
gcloud is the command-line tool for Google Cloud Platform. It comes pre-installed on Cloud Shell and supports tab-completion.
You can list the active account name with this command:
gcloud auth list
Credentialed accounts: - <myaccount>@<mydomain>.com (active)
Credentialed accounts: - firstname.lastname@example.org
You can list the project ID with this command:
gcloud config list project
[core] project = <project_ID>
[core] project = qwiklabs-gcp-44776a13dea667a6
Join Qwiklabs to read the rest of this lab...and more!
- Get temporary access to the Google Cloud Console.
- Over 200 labs from beginner to advanced levels.
- Bite-sized so you can learn at your own pace.
Create a new Cloud Storage bucket
Copy data files to the storage bucket