Overview

The purpose of a data request task is to create a historical API request, by date, for information from a source system. Submitting a task triggers our system to create a scheduled request to the source API for the defined period.

For example, you request Amazon Sponsored Advertising data for March 1st 2nd, 3rd, 4th, and 5th. There are 20 Amazon Sponsored Advertising data feeds. Each API request is done, by day, for a unique data feed (i.e., Sponsored Product Campaign, Sponsored Product Keywords, Sponsored Brands Campaigns.....). For 5 days and 20 data feeds, this will create 100 API requests (5 days x 20 data feeds).

Configuring A Task

Data request tasks are configured on a pipeline and source basis. A task is defined at a day level.

  • Start Date: The start date reflects the furthest date in the past you want to collect data from.

  • End Date: The end date is the most recent date you want data collected.

For example, the current date is April 30th. However, you want to create a task to collect data from April 1, 2022, to April 10, 2022. Since April 1 is the furthest date, this is when you want to "start" collecting data. You want to "end" collecting data on April 10.

If you want to collect data for one day, you would set the start and end dates to the same date. For example, a start date of April 13 and an end date of April 13.

Once submitted, it can take a number of days to complete the task. Why may it take so long? Prioritization, timing, and limits.

Prioritization

Data request tasks are run as secondary processes. This means it will take longer for this data to show up in your target destination. Why? Data request tasks are optimized as long-running background schedules, carefully requesting prior dates as API capacity permits. This minimizes impacts on existing go-forward daily, hourly, or lookback tasks already running.

Timing & Limits

The cadence and timing of requests are set by source system API limits, not Openbridge. For example, Amazon Advertising places a hard 60-day limit for historical data. While you may want data for the last two years, Amazon does not supply it. As such, we do not allow you to make requests for data that is not available or allowed by the source API.

In addition to hard limits imposed by APIs, there may be implied limits too. For example, Amazon Selling Partner API may not have a hard limit like Advertising but only allows 60 requests an hour. If you request 13 months of data for 7 report types, this will result in 2,700 reports (7 reports, by day, for 390 days). This can require close to 10,000 API requests for 13 months of data. This can take in excess of 4 weeks to complete given the limit of 60 requests per hour.

The Openbridge system is optimized to work within API limits, maximizing available API capacity. For example, with Amazon Selling Partner API we have to carefully sequence requests to the Amazon APIs to avoid the Amazon system throttling, which would slow any requests and responses by Amazon.

References

For more information on requests and scheduling, see https://docs.openbridge.com/en/articles/5675188-understanding-data-pipeline-scheduling

Note: Historical syncs are not available during a trial period.

Did this answer your question?