The Openbridge interface will present available datasets for a given connector. You can choose the ones you're most interested in. The number of available datasets will vary depending on your specific connector. Some connectors may offer a more extensive selection of datasets, while others may have fewer options, depending on the source API system's organization, capabilities, and limitations.
As you select datasets, watch the Load Index, which reflects the potential impact on the source system API performance.
Note: Dataset Definitions and Schemas: If you need more details, such as dataset descriptions or schemas, you can consult the Openbridge AI Data Copilot for help using the link provided.
Load Index Overview
The Load Index is based on the number of datasets selected and provides insight into how the system might handle your request.
The Load Index estimates the API load based on the number of datasets selected. It is determined by limits set by the source API, not Openbridge, meaning different connectors and APIs will have varying thresholds for acceptable load. As you select more datasets, the Load Index will adjust to reflect the projected load on the API based on its documented capacity.
Load Index Color Keys:
Green (Normal Load): Indicates regular, smooth operation with no expected delays.
Yellow (Low Load): Suggests mild load, which experiences potential minor delays.
Orange (High Load): Reflects a high load with API throttling, minor errors, retries, and delays.
Red (Severe Load): Signals a very high load, increasing error rates, longer retry queues, and significant API throttling.
Impact of Higher Load Levels (Orange/Red):
Reaching orange or red Load Index levels does not automatically mean your request will fail. Instead, it indicates that the requested data volume may push the boundaries of what the source API can handle in a reasonable timeframe.
Higher load levels typically imply that the system may take longer to process the request because the API could throttle the volume of data it allows in a given period. Here’s what this means:
Throttling and Queuing
The source API may throttle large quantities of requests, causing them to be delayed. If this happens, requests may be queued and retried after a few hours, resulting in longer processing times to land the data in a target destination. See Introduction To API Throttling
Retries
Although the system is designed to eventually fulfill the request, high load levels can increase latency or the time a dataset will appear in your destination. Depending on the source API's throttling policies and retry mechanisms, the total time required to retrieve and process the data may be extended. See API Backoff & Retry
Delays
While most requests will ultimately be successful, users should understand that the higher the Load Index, the more data delivery can experience some form of an error. This can delay data delivery as described in (1) and (2). For example, we document various use cases with Amazon SPAPI in Amazon Selling Partner API Data Feed Limits, Errors, And Constraints. Whether or not these delays are acceptable will depend on your specific use case or time sensitivity.
Considerations:
If your request is not time-sensitive, orange or red load levels may be acceptable, as the system will retry any throttled requests until successful.
If timing is critical, adjust your dataset selection to reduce the Load Index and ensure faster processing and minimal delay.
The Load Index shows the expected source API load level, allowing users to gauge the impact of their selected datasets on system performance. This ensures that users can make informed decisions about timing and data delivery.
Note: For more information on API limits, errors, throttling, retries, and much more, see these docs;
Summary
Users can optimize their data requests by selecting datasets while monitoring the Load Index to avoid system overloads and delays.
The Load Index helps ensure that users know the potential impact their request may have on the system, especially when working with larger datasets against the same API.
If users need more detailed information about datasets, such as schema details or descriptions, they can access the Openbridge AI Data Copilot for further assistance.