Skip to main content

Posts

Showing posts from January, 2021

Serverless - Unintentional Granularity Problems

  Serverless is great, but serverless functions rarely run by themselves... they're usually connected to data sources - and frequently that means SQL databases. In the graphic attached, my cloud SQL database is collapsing under assault by a serverless function that's scaling up instances to meet demand.  Here's how we unintentionally created this problem: To allow for parallel processing to speed processing, we've taken a transactional billing file, broken it into groups of 30 transactions, loaded those into messages and queued them.  We then have a serverless function that's listening for these events, and if messages are still waiting after a certain amount of time, another instance is spawned to begin handling the waiting messages and processing. When it was sent thousands of transactions, this worked great and the business users were suitably impressed with the processing speed (which dropped from many hours to a few minutes).  When they began sending larger tra