Historical Data Request Tasks

A data request task aims to initiate a retrospective API request, arranged by date, for information retrieval from a source system. When a task is submitted, our system is triggered to schedule a request to the source API for the specified duration.

For instance, would you like to get Amazon Sponsored Advertising data for March 1st through the 5th? Given the 20 different Amazon Sponsored Advertising data feeds, each day, a distinct API request is made for each unique data feed (such as Sponsored Product Campaign, Sponsored Product Keywords, Sponsored Brands Campaigns, etc.). So, over 5 days, 20 data feeds will generate 100 API requests (5 days multiplied by 20 data feeds).

Go to your Pipelines page in your Openbridge account. Select Request History in the menu options as shown:

You can then configure the historical data request task based on your specific needs, such as the dates and datasets.

Data request tasks are configured on a pipeline and source basis. A task is defined at a day level.

To help you understand Start and End dates, see below;

For instance, today is April 30, but you aim to establish a task to gather data from April 1, 2022, through April 12, 2022. You would set the "start" data collection date as the most recent date, April 12, and the "end" date as the earliest one, April 1.

If you're interested in data collection for a single day, the start and end dates would be set to that specific day, say April 13.

The Openbridge interface will present available datasets for a given connector. You can choose the ones you're most interested in. The number of available datasets will vary depending on your specific connector.

For more information on dataset selection and the Load Index, see API Load Index and Dataset Selection Guide.

Once a request has been made, your pipeline will show the process running in the background. If you cancel a request, it will reset any active or pending requests upstream at the source API.

Data request tasks operate as secondary processes. Consequently, the arrival of this data at your target destination might be delayed. This is because data request tasks are designed as prolonged background schedules that make requests for prior dates as API capacity allows. This approach ensures minimal impact on existing daily, hourly, or lookback tasks already in progress.

The rhythm and timing of requests are dictated by the limits of the source system's API, not by Openbridge. For example, Amazon Advertising has a strict 60-day boundary for historical data. So, even if you'd like data going back two years, Amazon won't provide it. Therefore, we don't allow requests for data that the source API cannot or does not permit.

In addition to these firm boundaries set by APIs, there might also be implicit limitations. For instance, Amazon's Selling Partner API may not have a strict limit like Advertising but only allows 60 requests per hour.

Requesting data for seven report types over 13 months will equate to 2,700 reports (7 reports each day, for 390 days). This would necessitate almost 10,000 API requests for 13 months of data and could take over four weeks to complete due to the 60 requests per hour limit.

The Openbridge system is designed to work within these API limitations, utilizing the available API capacity to the fullest. For instance, when using the Amazon Selling Partner API, we meticulously coordinate our requests to avoid Amazon's throttling system, which could otherwise slow down any requests and responses.

For more information on requests and scheduling, see https://docs.openbridge.com/en/articles/5675188-understanding-data-pipeline-scheduling