If you currently have or plan to have, a large number of data pipelines going to your Amazon Athena data lake, it is advised to raise your AWS service quotas from the default.

Amazon provides conservative defaults, which at scale, may create bottlenecks for those who have a large (30+) number of pipelines active, especially if those pipelines are high frequency (real-time or near-realtime).

While we will attempt to throttle and retry requests in the event rates are exceeded, it is possible that the volume of pipelines can far exceed the default limits on your AWS account. As a result, this may result in load errors as our attempts to deliver data eventually timeout.

The default quota for Athena APIs issues an error similar to the following:

For example, a quota message may look like this;

We strongly suggest creating CloudWatch monitors and alerts on your AWS account if you have not done so. In the event of a quota limit, AWS will alert you, and you will be able to take action.

If you encounter a limit, contact AWS to request a quota increase. Sign in to the AWS Management Console and open the Service Quotas console at https://console.aws.amazon.com/servicequotas/home.