Airbridge is an open-source project designed to enhance the Airbyte user experience. One of its requirements to successfully run Airbyte pipelines is the source and destination configuration specs.
However, configuration specifications for Airbyte connectors can be challenging to locate as they are often nested deep within the repository or spread across different directories.
Bring Your Own App
Open source connectors follow the "Bring Your Own App" model. A "Bring Your Own App" reflects a customer-created app, typically one for a data source system. For example, if you want to use the open-source Amazon Selling Partner API connector, Amazon requires you to "register" an app to access an account.
Once registered and approved, you will have access to the information needed to create a data source config file for the Amazon Selling Partner Airbyte connector.
How To Access Source and Destination Config Files?
Go to the Airbridge project on GitHub. In the project are two folders:
Data Sources: config/reference/source
Data Destinations: config/reference/destination.
Select the data source or destination configuration that matches your needs. You are welcome to browse the hundreds of pre-generated source and destination configs.
Configuration Examples
To provide a clearer understanding of what reference configs will look like, here are examples of configurations for a destination and a source:
Destination: destination_s3_destination_spec.json
You selected Amazon S3 as a data destination. The file can be found here:
This is what the config looks like:
{
"access_key_id": "optional_value",
"secret_access_key": "optional_value",
"s3_bucket_name": "required_value",
"s3_bucket_path": "required_value",
"s3_bucket_region": "required_value",
"format": {
"format_type": "optional_value",
"compression_codec": "optional_value",
"flattening": "optional_value",
"compression": {
"compression_type": "optional_value"
},
"block_size_mb": "optional_value",
"max_padding_size_mb": "optional_value",
"page_size_kb": "optional_value",
"dictionary_page_size_kb": "optional_value",
"dictionary_encoding": "optional_value"
},
"s3_endpoint": "optional_value",
"s3_path_format": "optional_value",
"file_name_pattern": "optional_value"
}
Source: source_tiktok_marketing_source_spec.json
You want to send TikTok data to your Amazon S3 destination. The config file for TikTok Marketing can be found here:
This is what the config looks like:
{
"credentials": {
"auth_type": "optional_value",
"app_id": "optional_value",
"secret": "optional_value",
"access_token": "optional_value",
"advertiser_id": "optional_value"
},
"start_date": "optional_value",
"end_date": "optional_value",
"attribution_window": "optional_value",
"include_deleted": "optional_value"
}
Important Note
The configurations extracted by Airbridge serve as reference points. They've been verified for their presence in the Airbyte GitHub repository, but users should verify before use due to the diverse nature of configurations.
While we utilized config.py to pre-generate configurations, users can also employ it as a stand-alone script, adjusting configurations on a case-by-case basis as their specific needs demand.