Why use AWS Lambda & SQS for managing large requests?
Recently, a scenario came in where a data of around 20,000 user had to be proceed. The data had to be calculated in the background and inserted into the DB after the calculation.
So, would hitting up with such large requests in a production server work? What if the server crashes, slow down in the middle of the operation?
Here comes AWS lambda to the rescue. We deployed our code on AWS lambda and this eliminated our hassle of taking caring of server. Also, another reason of choosing AWS lambda was the cost since it calculates the server cost based on resources and the time the server runs.
Also, Triggering AWS lambda for every requests(20,000) could exhaust the server .Something that could store all the request, very secure,reliable and send it to lambda function on request could solve our issue.
Amazon SQS was the solution to the problem. SQS basically stores the data in queue and could send it to lambda. A certain batch of data is request by lambda and SQS sends its upon the request.
So, the basic architecture for the solution we planned is a below:
The server retrieves all the info needed for lambda to calculate the required information and sends it to SQS. SQS stores the data and send it to lambda for the calculation & the values were inserted to the Database.Without over burden in the server , we could execute large information in over few minutes.
The below stats show lambda handling up to 454 requests with no error rates. Lambda could easily handle large number of request with no downtime.