Airbridge: Overview Of Airbyte Data Flows

Airbridge orchestrates data ingestion pipelines from Airbyte data sources like Stripe, Facebook, or Google to Airbyte data destination connectors like S3 data lake, Redshift, Snowflake, or BigQuery.

Airbridge is a configuration-driven, Airbyte connector service. It is open source, released under the MIT license. For more detail on Airbridge, visit the project on Github: https://github.com/openbridge/airbridge

For docs on activating Airbridge and Airbyte on your AWS account, please see Activating Airbridge In Your AWS Account.

Understanding Airbyte Connector Configurations

A core feature of an open source Airbyte Cloud runtime for a data pipeline is the “connector .” Airbyte has two classes of connectors: data source and destination. These connector acts as a bridge, facilitating data movement between various platforms. Both the source and destination require "configs" to connect, authorize, and process data.

Airbyte Data Source Config

The term "data source" in the context of Airbyte refers to the original location or platform from which data is extracted. This could be a CRM system, web analytics, databases, or any other platform where raw data is generated and stored.

Here is an example of a klaviyo soruce config which requires an API key and date:

{
    "api_key": "required_value",
    "start_date": "required_value"
}

However, you can see the variation in the salesforce source config requirements;

{
    "is_sandbox": "optional_value",
    "auth_type": "optional_value",
    "client_id": "required_value",
    "client_secret": "required_value",
    "refresh_token": "required_value",
    "start_date": "optional_value",
    "force_use_bulk_api": "optional_value",
    "streams_criteria": "optional_value"
}

The respective configs are defined by the source itself.

Scheduling and Timing When Collecting Data From A Source

While you may want "live", real-time data, the data source system defines when data is available and the frequency it can be requested.

Scheduling defines the cadence at which the data pipeline will run. Data source APIs are not unlimited resources; they have restrictions. If you set a schedule that exceeds the capacity, the source API will block, throttle, or fail your requests. A schedule should only be based on recommended frequencies for a given data source.

For more information on scheduling, Understanding Data Source APIs

Airbyte Source Catalogs

One of the key components of Airbyte's configuration is the catalog.json file, which defines the streams (i.e., tables, collections, or any other structured data) that a specific source connector can read from. This file plays an integral role in the ETL (Extract, Transform, Load) process, as it defines the schema for the data that will be extracted from a source and loaded into a destination.

To find the catalog.json, you will need to navigate to the respective sources on Github. For example, you were interested in Chargebee, go to source-chargebee/integration_tests/. In that folder, you would find the configured_catalog.json. Typically, no changes are needed to the catalog unless you are familiar with customizing this file. For most, leaving it as-is would be sufficient.

Airbyte Data Destinations Config

The term "data destination" in the context of Airbyte Cloud refers to where the integrated data will land. This could be a specific cloud database, data warehouse, data lake, or another storage platform.

Like sources, destinations also have configs. The following is an example config for Amazon Web Services S3:

{
    "access_key_id": "optional_value",
    "secret_access_key": "optional_value",
    "s3_bucket_name": "required_value",
    "s3_bucket_path": "required_value",
    "s3_bucket_region": "required_value",
    "format": {
        "format_type": "optional_value",
        "compression_codec": "optional_value",
        "flattening": "optional_value",
        "compression": {
            "compression_type": "optional_value"
        },
        "block_size_mb": "optional_value",
        "max_padding_size_mb": "optional_value",
        "page_size_kb": "optional_value",
        "dictionary_page_size_kb": "optional_value",
        "dictionary_encoding": "optional_value"
    },
    "s3_endpoint": "optional_value",
    "s3_path_format": "optional_value",
    "file_name_pattern": "optional_value"
}

References

The following is a reference collection of data source documentation. This is not meant to be a comprehensive list, merely a waypoint to help get people pointed in the right direction.

Connector Name	Documentation Page
Postgres	Postgres
ActiveCampaign	ActiveCampaign
Adjust	Adjust
Aha API	Aha API
Aircall	Aircall
Airtable	Airtable
AlloyDB for PostgreSQL	AlloyDB for PostgreSQL
Alpha Vantage	Alpha Vantage
Amazon Ads	Amazon Ads
Amazon Seller Partner	Amazon Seller Partner
Amazon SQS	Amazon SQS
Amplitude	Amplitude
Apify Dataset	Apify Dataset
Appfollow	Appfollow
Apple Search Ads	Apple Search Ads
AppsFlyer	AppsFlyer
Appstore	Appstore
Asana	Asana
Ashby	Ashby
Auth0	Auth0
AWS CloudTrail	AWS CloudTrail
Azure Blob Storage	Azure Blob Storage
Azure Table Storage	Azure Table Storage
Babelforce	Babelforce
Bamboo HR	Bamboo HR
Baton	Baton
BigCommerce	BigCommerce
BigQuery	BigQuery
Bing Ads	Bing Ads
Braintree	Braintree
Braze	Braze
Breezometer	Breezometer
CallRail	CallRail
Captain Data	Captain Data
Cart.com	Cart.com
Chargebee	Chargebee
Chargify	Chargify
Chartmogul	Chartmogul
ClickHouse	ClickHouse
ClickUp API	ClickUp API
Clockify	Clockify
Close.com	Close.com
CockroachDB	CockroachDB
Coda	Coda
CoinAPI	CoinAPI
CoinGecko Coins	CoinGecko Coins
Coinmarketcap API	Coinmarketcap API
Commcare	Commcare
Commercetools	Commercetools
Configcat API	Configcat API
Confluence	Confluence
ConvertKit	ConvertKit
Convex	Convex
Copper	Copper
Courier	Courier
Customer.io	Customer.io
Datadog	Datadog
DataScope	DataScope
Db2	Db2
Delighted	Delighted
Dixa	Dixa
Dockerhub	Dockerhub
Dremio	Dremio
Drift	Drift
Drupal	Drupal
Display & Video 360	Display & Video 360
Dynamodb	Dynamodb
End-to-End Testing Source for Cloud	End-to-End Testing Source for Cloud
End-to-End Testing Source	End-to-End Testing Source
Elasticsearch	Elasticsearch
EmailOctopus	EmailOctopus
Everhour	Everhour
Exchange Rates API	Exchange Rates API
Facebook Marketing	Facebook Marketing
Facebook Pages	Facebook Pages
Faker	Faker
Fastbill	Fastbill
Fauna	Fauna
Files (CSV, JSON, Excel, Feather, Parquet)	Files (CSV, JSON, Excel, Feather, Parquet)
Firebase Realtime Database	Firebase Realtime Database
Firebolt	Firebolt
Flexport	Flexport
Freshcaller	Freshcaller
Freshdesk	Freshdesk
Freshsales	Freshsales
Freshservice	Freshservice
FullStory	FullStory
Gainsight-API	Gainsight-API
GCS	GCS
Genesys	Genesys
getLago API	getLago API
GitHub	GitHub
GitLab	GitLab
Glassfrog	Glassfrog
GNews	GNews
GoCardless	GoCardless
Gong	Gong
Google Ads	Google Ads
Google Analytics 4 (GA4)	Google Analytics 4 (GA4)
Google Analytics (Universal Analytics)	Google Analytics (Universal Analytics)
Google Directory	Google Directory
Google PageSpeed Insights	Google PageSpeed Insights
Google Search Console	Google Search Console
Google Sheets	Google Sheets
Google-webfonts	Google-webfonts
Google Workspace Admin Reports	Google Workspace Admin Reports
Greenhouse	Greenhouse
Gridly	Gridly
Gutendex	Gutendex
Harness	Harness
Harvest	Harvest
HTTP Request	HTTP Request
Hubplanner	Hubplanner
HubSpot	HubSpot
Insightly	Insightly
Instagram	Instagram
Instatus	Instatus
Intercom	Intercom
Intruder.io API	Intruder.io API
Ip2whois API	Ip2whois API
Iterable	Iterable
Jenkins	Jenkins
Jira	Jira
K6 Cloud API	K6 Cloud API
Kafka	Kafka
Klarna	Klarna
Klaviyo	Klaviyo
Kustomer	Kustomer
Kyriba	Kyriba
Kyve Source	Kyve Source
Launchdarkly API	Launchdarkly API
Lemlist	Lemlist
Lever Hiring	Lever Hiring
LinkedIn Ads	LinkedIn Ads
LinkedIn Pages	LinkedIn Pages
Linnworks	Linnworks
Lokalise	Lokalise
Looker	Looker
Magento	Magento
Mailchimp	Mailchimp
MailerLite	MailerLite
Mailersend	Mailersend
MailGun	MailGun
Mailjet - Mail API	Mailjet - Mail API
Mailjet - SMS API	Mailjet - SMS API
Marketo	Marketo
Merge	Merge
Metabase	Metabase
Microsoft Dataverse	Microsoft Dataverse
Microsoft Dynamics AX	Microsoft Dynamics AX
Microsoft Dynamics Customer Engagement	Microsoft Dynamics Customer Engagement
Microsoft Dynamics GP	Microsoft Dynamics GP
Microsoft Dynamics NAV	Microsoft Dynamics NAV
Microsoft Teams	Microsoft Teams
Mixpanel	Mixpanel
Monday	Monday
Mongo DB	Mongo DB
Microsoft SQL Server (MSSQL)	Microsoft SQL Server (MSSQL)
My Hours	My Hours
MySQL	MySQL
N8n	N8n
NASA	NASA
Oracle Netsuite	Oracle Netsuite
News API	News API
Newsdata API	Newsdata API
Notion	Notion
New York Times	New York Times
Okta	Okta
Omnisend	Omnisend
OneSignal	OneSignal
Open Exchange Rates	Open Exchange Rates
OpenWeather	OpenWeather
Opsgenie	Opsgenie
Oracle Peoplesoft	Oracle Peoplesoft
Oracle Siebel CRM	Oracle Siebel CRM
Oracle DB	Oracle DB
Orb	Orb
Orbit	Orbit
Oura	Oura
Outreach	Outreach
PagerDuty	PagerDuty
Pardot	Pardot
Partnerstack	Partnerstack
Paypal Transaction	Paypal Transaction
Paystack	Paystack
Pendo	Pendo
PersistIq	PersistIq
Pexels-API	Pexels-API
Pinterest	Pinterest
Pipedrive	Pipedrive
Pivotal Tracker	Pivotal Tracker
Plaid	Plaid
Plausible	Plausible
Pocket	Pocket
PokéAPI	PokéAPI
Polygon Stock API	Polygon Stock API
Postgres	Postgres
PostHog	PostHog
Postmarkapp	Postmarkapp
PrestaShop	PrestaShop
Primetric	Primetric
Public APIs	Public APIs
Punk-API	Punk-API
PyPI	PyPI
Qonto	Qonto
Qualaroo	Qualaroo
QuickBooks	QuickBooks
Railz	Railz
RD Station Marketing	RD Station Marketing
Recharge	Recharge
Recreation.gov API	Recreation.gov API
Recruitee	Recruitee
Recurly	Recurly
Redshift	Redshift
Reply.io	Reply.io
Retently	Retently
RingCentral	RingCentral
Robert Koch-Institut Covid	Robert Koch-Institut Covid
Rocket.chat API	Rocket.chat API
RSS	RSS
S3	S3
Salesforce	Salesforce
Salesloft	Salesloft
SAP Business One	SAP Business One
sap-fieldglass	sap-fieldglass
SearchMetrics	SearchMetrics
Secoda API	Secoda API
Sendgrid	Sendgrid
Sendinblue API	Sendinblue API
Senseforce	Senseforce
Sentry	Sentry
SFTP Bulk	SFTP Bulk
SFTP	SFTP
Shopify	Shopify
Shortio	Shortio
Slack	Slack
Smaily	Smaily
SmartEngage	SmartEngage
Smartsheets	Smartsheets
Snapchat Marketing	Snapchat Marketing
Snowflake	Snowflake
Sonar Cloud API	Sonar Cloud API
SpaceX-API	SpaceX-API
Spree Commerce	Spree Commerce
Square	Square
Statuspage.io API	Statuspage.io API
Strava	Strava
Stripe	Stripe
Sugar CRM	Sugar CRM
SurveySparrow	SurveySparrow
SurveyCTO	SurveyCTO
SurveyMonkey	SurveyMonkey
Talkdesk Explore	Talkdesk Explore
Tempo	Tempo
Teradata	Teradata
The Guardian API	The Guardian API
TiDB	TiDB
TikTok Marketing	TikTok Marketing
Timely	Timely
TMDb	TMDb
Todoist	Todoist
Toggl API	Toggl API
TPL/3PL Central	TPL/3PL Central
Trello	Trello
TrustPilot	TrustPilot
TVMaze Schedule	TVMaze Schedule
Twilio Taskrouter	Twilio Taskrouter
Twilio	Twilio
Twitter	Twitter
Tyntec SMS	Tyntec SMS
Typeform	Typeform
Unleash	Unleash
US Census API	US Census API
Vantage API	Vantage API
VictorOps	VictorOps
Visma e-conomic	Visma e-conomic
Vittally	Vittally
Waiteraid	Waiteraid
Weatherstack	Weatherstack
Webflow	Webflow
Whisky Hunter	Whisky Hunter
Wikipedia Pageviews	Wikipedia Pageviews
WooCommerce	WooCommerce
Wordpress	Wordpress
Workable	Workable
Workramp	Workramp
Wrike	Wrike
Xero	Xero
XKCD	XKCD
Yahoo Finance Price	Yahoo Finance Price
Yandex Metrica	Yandex Metrica
Yotpo	Yotpo
Younium	Younium
YouTube Analytics	YouTube Analytics
Zapier Supported Storage	Zapier Supported Storage
Zencart	Zencart
Zendesk Chat	Zendesk Chat
Zendesk Sell	Zendesk Sell
Zendesk Sunshine	Zendesk Sunshine
Zendesk Support	Zendesk Support
Zendesk Talk	Zendesk Talk
Zenefits	Zenefits
Zenloop	Zenloop
Zoho CRM	Zoho CRM
Zoom	Zoom
Zuora	Zuora