Consider the following scenario: there is a source table for "Product" named
product_v1. Openbridge automatically generates an aggregate view for all source tables called
product_master . To support downstream ETL or analytics efforts you create a custom view called
my_product_view that is based on the
A few weeks after
my_product_view there is an upstream change to Product that changes to the underlying data structures. In the Openbridge system, this triggers the automated creation of a new table version to adjust to the change. In this example, a new source table called
product_v2 is created . In addition to the new table version, the system will also automatically update
product_master view to aggregate two source tables; the older
product_v1 and the new
This creates a challenge for the custom
my_product_view. Databases systems like Amazon Redshift or Google BigQuery will require that you first drop all associated views before creating a new one. This means that in order to replace
product_master views must be dropped. In-place substitution is not permitted by those systems. However, once
product_master is dropped, all other reliant views, including
my_product_view, will be dropped as well. This is where a break can occur for custom views like
my_product_view. While the Openbridge system automates change management, more often than not a customer is not backing up their custom views or if they are, they are not able to restore a view prior to something breaking. The loss of
my_product_view can disrupt ETL, analytics or other data engineering activities.
Our solution is to automatically create a snapshot of all of your views. How does it work?
We will regularly backup the SQL used to create all views on the database you activated with Openbridge. This process will run once a day at 10 AM UTC. In the event there is a migration that needs to occur, our auto migration process will verify that all affected views have backups available. In most cases, this will be true since we are automatically backing up the table and view definitions.
However, if we detect that a backup is missing, the migration process will halt and we’ll be notified that some views are not backed up. We will then fix this for you. If backups are available then the auto-migration will continue we will restore your custom views after the process is completed.
Our backup may not match your current custom view if it changed after our last backup. For example, we backup
my_product_view at 2 AM EST. You make changes to
my_product_view at 9 AM EST. An auto-migration is triggered by changes to source data at 1 PM EST. We drop
my_product_view as part of the auto-migration and restore with our backup. However, the backup reflects
my_product_view as of 2 AM EST, not 9 AM EST. This means we will restore an older version of your view.
Normally views are slowly changing, so the risk of this use case is low. However, we recommend that you keep backups of your work as a fail-safe outside of the automated process we undertake on your behalf. In the future, the backup process will be automatically triggered prior to an auto-migration