Google describes BigQuery as “an enterprise data warehouse that provides super-fast SQL queries using the processing power of Google’s infrastructure.” Openbridge supports delivery of data to a BigQuery warehouse destination. If you are not a BigQuery user, Google does offer a free tier which is perfect to kickstart your data efforts.
There are a few steps that you will need to care for before we can get BigQuery setup within Openbridge. We need to get your Google Cloud configuration setup. Specifically:
- You need to select the project you want to use within your Google Cloud Console. This is typically a drop down menu in the Google Cloud console nav
- You need to enable your BigQuery API for the selected project
- You need to create a Service Account and IAM policy that allows Openbridge to access to BigQuery within your project
The process should take no more than 5 minutes. Lets get started!
Step 1: Enable The BigQuery API
The first step is to verify that you have the BigQuery API enabled. If not, then enable it.
p.s. Make sure you have select the correct project in the nav
Step 2: Create Service Account
The next step is to configure a service account via IAM. You need to login to your Google Cloud console:
After you login select "IAM & admin" in the navigation panel.
Next, you want to select "Service accounts" and then "Create service account"
This will pop open the create service account window.
- A: Give your account a unique name
- B: Copy the "Service Account ID" as you will need this later. It will look something like this "firstname.lastname@example.org"
- C: Select "JSON" as the Key type
p.s. Don't select a Role, we will do that later.
When you are ready select "CREATE"
Your service account JSON will download to your computer. The content of the file will look something like this:
Keep this file in a safe place. Treat it like you would a password. You will need this later when setup BigQuery within Openbridge.
You should see your newly created Service Account listed:
Step 3: Adding Permissions To Your Service Account
Next, we need to add BigQuery permissions to your new service account.
In your Google Cloud console select "IAM & admin"->"IAM"
You will see the "ADD" option. This will allow you to set permissions for your new Service Account.
You will be presented a new window:
- A: Remember the Service Account ID we asked you to save? You enter it here. Paste the Service Account ID where is says "New members"
- B: Select the BigQuery Admin role.
Google also requires Project Browser role be set. If the Project Browser is not set it will result in an error due to the Google Cloud projects being empty.
As a result both BigQuery Admin and Project Browser must be set!
When you are done, select "SAVE"
Your new member and permissions for your BigQuery Project should be listed:
Congrats! You just configured a Service account that will allow us to deliver data to a BigQuery warehouse destination. With your Google account configured, you can now add a BigQuery warehouse destination within Openbridge.
One Last Thing...Google Cloud Billing!
Some Google Cloud projects do not have billing enabled. You need to make sure your target project has it enabled. Without billing being active Google prevents us from being able to connect to BigQuery!
First, go to "Billing" in the Google Cloud Console
Next, you will want to make sure Billing is active and configured. If it is not, turn Billing on.
Here are reference docs from Google about setting up billing: