Google describes BigQuery as “an enterprise data warehouse that provides super-fast SQL queries using the processing power of Google’s infrastructure.” Openbridge supports the delivery of data to a BigQuery warehouse destination. If you are not a BigQuery user, Google does offer a free tier which is perfect to kickstart your data efforts.
There are a few steps that you will need to care for before we can get BigQuery set up within Openbridge. We need to get your Google Cloud configuration setup. Specifically:
You need to select the project you want to use within your Google Cloud Console. This is typically a drop-down menu in the Google Cloud console nav
You need to enable your BigQuery API for the selected project
You need to create a Service Account and IAM policy that allows Openbridge to access to BigQuery within your project
The process should take no more than 5 minutes. Let's get started!
Step 1: Enable The BigQuery API
The first step is to verify that you have the BigQuery API enabled. If not, then enable it.
p.s. Make sure you have selected the correct project in the nav
Step 2: Create Service Account
The next step is to configure a service account via IAM. You need to login to your Google Cloud console:
After you log in select "IAM & admin" in the navigation panel.
Next, you want to select "Service Accounts" and then "Create service account"
This will pop open the create service account window.
A: Give your account a unique name
B: Copy the "Service Account ID" as you will need this later. It will look something like this "firstname.lastname@example.org"
C: Select "JSON" as the Key type
p.s. Don't select a Role, we will do that later.
When you are ready select "CREATE"
Your service account JSON will download to your computer. The content of the file will look something like this:
Keep this file in a safe place. Treat it like you would a password. You will need this later when setup BigQuery within Openbridge.
You should see your newly created Service Account listed:
Step 3: Adding Permissions To Your Service Account
Next, we need to add BigQuery permissions to your new service account.
In your Google Cloud console select "IAM & admin"->"IAM"
You will see the "ADD" option. This will allow you to set permissions for your new Service Account. Select "ADD"
You will be presented with a new window:
A: Remember the Service Account ID we asked you to save? You enter it here. Paste the Service Account ID where it says "New members"
B: Select the BigQuery Admin role.
NOTE: Google also requires the Project Browser role to be set. If the Project Browser is not set it will result in an error due to the Google Cloud projects being empty. As a result, both BigQuery Admin and Project Browser must be set!
When you are done, select "SAVE"
Your new member and permissions for your BigQuery Project should be listed:
Congrats! You just configured a Service account that will allow us to deliver data to a BigQuery warehouse destination.
One Last Thing...Google Cloud Billing!
Some Google Cloud projects do not have billing enabled. You need to make sure your target project has it enabled. Without billing being active Google prevents us from being able to connect to BigQuery!
First, go to "Billing" in the Google Cloud Console:
Next, you will want to make sure Billing is active and configured. If it is not, turn Billing on.
Here are reference docs from Google about setting up billing: