If you currently have or plan to have, a large number of data pipelines going to your Amazon Athena data lake, it is advised to raise your AWS service quotas from the default.
Amazon provides conservative defaults, which at scale, may create bottlenecks for those who have a large (30+) number of pipelines active, especially if those pipelines are high frequency (real-time or near-realtime).
While we will attempt to throttle and retry requests in the event rates are exceeded, it is possible that the volume of pipelines can far exceed the default limits on your AWS account. As a result, this may result in load errors as our attempts to deliver data eventually timeout.
Amazon Athena Quota Error Examples
The default quota for Athena APIs issues an error similar to the following:
"ClientError: An error occurred (ThrottlingException) when calling the <API_name> operation: Rate exceeded."
For example, a quota message may look like this;
"You have exceeded the limit for the number of queries you can run concurrently. Please reduce the number of concurrent queries submitted by this account. Contact customer support to request a concurrent query limit increase.""Reason":"CONCURRENT_QUERY_LIMIT_EXCEEDED"
Setting Athena Service Monitors and Service Alerts
We strongly suggest creating CloudWatch monitors and alerts on your AWS account if you have not done so. In the event of a quota limit, AWS will alert you, and you will be able to take action.
Requesting Amazon Athena Quota Increase
If you encounter a limit, contact AWS to request a quota increase. Sign in to the AWS Management Console and open the Service Quotas console at https://console.aws.amazon.com/servicequotas/home.
In the navigation pane, choose AWS services.
Choose the Athena service from the list, or type the name of the Athena service in the search box.
If the quota is adjustable, you can choose the button or the name, and then choose Request quota increase.
For the Change quota value, enter the new value. The new value must be greater than the current value.