
> Recently, I spoke at two meetups on **”Unlocking Serverless Observability”**, which is an important topic, so I have decided to publish an article about the demo project I put together to help illustrate - When Serverless Goes Wrong.

# A Working Sample

Supporting my recent talks and this article is a working (and non-working) example of an Event Driven data integration Architecture that can be [found on my GitHub page](https://github.com/walmsles/serverless-observability).  The README is comprehensive and explains how to install and run the performance tests if you want to see it yourself.

 **WARNING: Executing a performance test even with AWS Lambda may incur charges on your AWS Account.  Given the generous free tier, this is unlikely, but you have been warned.**

# About the Project

Alice works for Widgets Incorporated, a popular online retailer of technology widgets.  Alice keeps the lights on for the e-commerce website running on a large EC2 Instance on the AWS cloud.  Alice’s most important job is ensuring the weekly sales export runs for the sales and marketing team.

![Meet Alice](/strands/observability/when-serverless-goes-wrong/meet-alice.png)

One day, Max from sales and marketing seeks out Alice and demands something to be done about making sales data available more than just weekly!  In the same conversation, he also shares that they are changing delivery providers and they will need to integrate into Ace Courier’s scheduling API to coordinate deliveries.

![Meet Max from Marketing who is very angry and gruff](/strands/observability/when-serverless-goes-wrong/meet-max.png)

Alice is very worried about adding more processing to the existing e-commerce platform.  It is at capacity, and Widgets Inc. doesn’t have a large budget to do much more about it!  Alice digs through documentation wikis and stack overflow, trying to think how this problem can be resolved.  When she was ready to give up, someone replied to her question about stack overflow.  Widgets Inc.’s e-commerce solution has a lesser-known notification webhook feature that can be turned on through configuration.  

![Alice Searches for a Solution](/strands/observability/when-serverless-goes-wrong/what-can-alice-do.png)

Excitedly she turns to [Serverless Land](https://serverlessland.com) to learn more about Event Driven Architecture so she can build a serverless event hub to handle live order notifications.  With this simple configuration change, she knows she can deliver live sales data for Max in Sales and Marketing and do a live integration with Ace Courier’s scheduling API.

![Event Driven Architectures on ServerlessLand.com](/strands/observability/when-serverless-goes-wrong/eda-serverlessland.png)

On Serverless Land, Alice finds a GitHub reference to an Event Driven pattern which will be useful to solve her exact problem.  The [pattern she found on Serverless Land](https://serverlessland.com/patterns/apigw-lambda-eventbridge-sam-java) is for Java, but that is okay because the existing sam template will accelerate the IaC aspect of building out the serverless event hub in Python.

![Event Driven Architectures on ServerlessLand.com](/strands/observability/when-serverless-goes-wrong/eda-pattern.png)

Alice has been learning about Serverless with Python and has been reading about [SAM CLI](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/what-is-sam.html) and [AWS Lambda Powertools for Python](https://awslabs.github.io/aws-lambda-powertools-python).

![Event Driven Architectures on ServerlessLand.com](/strands/observability/when-serverless-goes-wrong/widget-inc-eda.png)

# How Alice Built it?

Alice used AWS Lambda power tools for logging and tracing and simplifying her code.  For the Delivery API Lambda, she used the [tenacity library](https://pypi.org/project/tenacity/), which is great for adding retries to any function in Python to ensure the lambda recovered from any failures.

![Sample code for calling Ace Couriers API using tenacity library for retries](/strands/observability/when-serverless-goes-wrong/delivery-api-code.png)

Alice used AWS Lambda Powertools for Python which has an excellent structured logger.  She made good use of this and set the `service` name to enable logs to be grouped by each service processing data.  Alice leveraged the `correlation_id_path` log setting, which enables json-path extraction of attribute values from the AWS Event being processed to automatically populate the `correlation_id` so she could trace transactions across the entire distributed stack.  Alice also listened to my [Getting Started](/strands/observability/getting-started) tips by making sure the logs tell a story and using the `START`, `COMPLETE` and `FAILED` status values as a `status` JSON attribute for key logs to assist in tracing.

![Sample code for calling Ace Couriers API using tenacity library for retries](/strands/observability/when-serverless-goes-wrong/delivery-handler-code.png)

Alice did quite well and completed the build reasonably quickly - she had good logging and retries for API integration failures which are all great considerations for building a resilient event-driven architecture.

# So What Went Wrong?

One of the key things Alice didn’t consider in her build, which is common for many teams I work with, is understanding the transaction Volumes that the architecture will be processing.  This is critical to understand since Serverless is hyper-scalable and will handle almost any traffic you throw at it!  Alice also didn’t consider the potential limits of the external Courier API she was interacting with - understanding the constraint of ALL your downstream systems is just as critical!  If you know your volumes and downstream limits - you will quickly learn up-front at design time whether any scalability boundaries will be problematic.

# When Serverless Goes Wrong

To demonstrate the problem when Scalability Boundaries clash, I built Alice’s demo architecture and introduced a deliberate Scalability boundary through a **slow-api** service representing Ace Couriers.

## The Demo Project Constraints

There are 3 components to this project:

1. **NotificationFunction** - this takes the payload from the API, injects a correlation_id and forwards it to EventBridge as a notify-order event.
2. **DeliveryFunction** - This is a simple function that takes the detail of the event, removes the meta-data from the body and sends this to the configured API endpoint. This lambda uses the [tenacity library](https://pypi.org/project/tenacity/) for retries in the code with a wait time between 3 and 8 seconds on any failure from calling the “slow API”.
3. **SlowAPI** - This is an API that has been set up to mimic an unstable real-world API and has been setup with the following constraints:

* 20% of the API Calls will fail immediately.
* The API will take between 0 and 2 seconds to return a response when it runs successfully.

The **SlowApi** lambda is triggered by an API Gateway route with an API Key and usage plan set to restrict the number of transactions through it to 10 per second: BurstLimit = 10, RateLimit = 10. The **NotificationHandler** is also triggered by an API Gateway with no API Key and no rate limiting.

There is a configured artillery test file that can be used to run a performance test with the following characteristics:

* For 1 minute test will hit the Notification API at approx. 2 transactions per second
* For the next 2 minutes, transactions will ramp up to 5 transactions per second
* For the next 10 minutes, transactions will ramp from 4 per second to 15 per second and then remain at this level which is a level that exceeds the SlowAPI transaction limit.

There are 2 branches on the demo project - `no-flow-control` is what Alice built.  `flow-control` is the same architecture but with flow control introduced to ensure the event integration flow is controlled.

## What Happened During the Performance Test?

The Slow API worked quite well - it had errors, but these were planned errors, approximately 20% of all requests.  So the graph below is a fairly solid representation of a 20% error rate with the green line wavering around the 80% success mark as planned.  Nice to see part of the system worked!

![Metric graph for the slow-API handler](/strands/observability/when-serverless-goes-wrong/slow-api-no-flow.png)

As you can see from the **Delivery Lambda Success** dashboard graph below, the Delivery Lambda was fine for the first seven (7) minutes before the success rate started to drop.  What is impressive from a Lambda perspective is that at the 8-minute mark, we have a success rate greater than 90%, whilst we have errors sitting at nearly 50 in that minute.  This speaks to the automatic resilience that is built into the AWS Lambda asynchronous invocation mechanism.  Sadly as time continues, the success rate bottoms out at approximately 20% at the 17-minute mark with a peak error rate of 1535 failed lambda invocations.  This graph picture is something you will see a LOT on the metrics dashboard for your lambda functions when errors are occurring.  The important thing to look at is the bottom number of the range for the % Success (where it bottoms out) since it may bottom out at 97%, which from an overall system health perspective, is still quite healthy.

![Delivery handler success without flow control](/strands/observability/when-serverless-goes-wrong/delivery-success-no-flow.png)

When you have scalability boundaries clashing, what you will see is a steady growth in your overall Lambda Concurrency metric, as shown below.  This graph peaks at 788, just shy of my account level limit of 1,000 concurrent invocations.  A common reaction as people see this is to urgently reach out to your AWS Technical Account Manager and very quickly increase the account concurrency so that you have more headroom.  Unfortunately, when you have this reaction, you only create a larger pool of concurrency for the out-of-control lambda function to draw from, which will make the graph peak much larger and effectively make your bill slightly larger.

![Account Concurrency metric graph showing steady growth over time](/strands/observability/when-serverless-goes-wrong/concurrency-no-flow.png)

When you have no flow control built into your system, there is nothing much you can do at this point but ride out the storm or deactivate lambda function triggers to stop your system from creating bigger problems for the downstream systems.  This is the key architectural component missing for this event-driven architecture - **Flow Control**.  The overall invocation count metric graph tells a very similar story, as expected.

![Invocation count metric graph](/strands/observability/when-serverless-goes-wrong/invocations-no-flow.png)

## Why Did this happen?

The scale madness happens because we have a Scalability Boundary between the Delivery API Lambda and the Ace Courier System.  Anytime we have a hyper-scalable system talking to a less scalable system or a system of unknown scale, this is a Scalability Boundary. This is where scale differences occur and cause problems.  As Serverless Architects, we need to be able to identify these boundaries and design our architectures to ensure our system remains resilient and working at all times.  This boundary can also be considered a domain boundary. We must know all interactions beyond our bounded context and manage failure carefully to avoid losing data due to a **Retry Storm**, which is what causes the graph peaks and high concurrency load.

![image showing uncontrolled scale in the architecture](/strands/observability/when-serverless-goes-wrong/scalability-boundary.png)

## How to fix

To resolve scalability boundary clashes, we need to add a flow control buffer that can provide a retry mechanism on failure.  The best example is using an SQS queue as the buffer element coupled with maximum concurrency configuration on the Lambda trigger’s Event Source Mapping.  This combination is the most powerful flow control mechanism available on AWS today.

![Architecture diagram showing flow control](/strands/observability/when-serverless-goes-wrong/eda-flow-control.png)

## How does Flow Control help?

The slow API in our system with flow control is behaving as expected.  The success rate is wavering around the 80% mark by design.  The error rates are much lower - this is because of the introduction of flow control, which controls the number of concurrent calls into the Slow API endpoint.  As you can see, the Slow API (or Ace Couriers System) is much calmer and not spiraling out of control like before.  A graph like this is not ideal because there are errors, but it’s not showing a system that is completely out of control, so there is time to work to correct the problem.  In this case, the error rate is because we coded it in, but in a real system with transient errors like this, we need to look at logging to determine the issue and then deploy a fix.

![Slow API success metric graph](/strands/observability/when-serverless-goes-wrong/slow-api-flow-control.png)

The **Delivery API Success Graph** shows the switching of success to error and back again - but notice the lowest value on the Y-axis is around the 98% mark.  So although there are errors, it is not disastrous (yet).

![Delivery API Success metric graph](/strands/observability/when-serverless-goes-wrong/delivery-success-flow-control.png)

With flow control in place, the concurrent lambda execution is way different - it is a very steady flat line which is what we expect to see in a healthy system.  It is hovering around a total concurrency of 20 across the entire system, which is very healthy!

![Account Concurrency metric graph](/strands/observability/when-serverless-goes-wrong/concurrency-flow-control.png)

Invocations are also okay, as you can see from the graph.

![Account Invocation metric graph](/strands/observability/when-serverless-goes-wrong/invocations-flow-control.png)

In these last 2 graphs, you can see 2 anomalous spikes.  Something weird happened with the metrics for the Notification function. At 13:55, cloud watch stopped recording metrics for this function and instead recorded all counts in 2 batches at 14:12 and again at 14:28.  Outside of these 2 anomalies, you can notice the steady, even pace at which the lambda invocations happen through the flow control mechanism provided by the SQS Queue.

## Why is Structured Logging so Important

One of the great things Alice did well was to use AWS Lambda Powertools for the Logging utility, which emitted structured logs in JSON structure.  Part of the logging utility enables capturing the transaction **correlation_id** along with function details from the lambda context and a defined **service** name, like the entry shown below.

![Structured logging sample log message](/strands/observability/when-serverless-goes-wrong/structured-log.png)

The additional fields enable Alice to search for everything that happened to a single transaction to **Tell the Story** of what happened.  An example output of a correlation_id story is shown below:

![Log query showing a transaction story](/strands/observability/when-serverless-goes-wrong/log-story.png)

A feature I get all my teams to do is to emit a **START**, **COMPLETED**, and **FAILED** log message as a status.  This is powerful to query logs over a time frame to determine whether everything was processed correctly or not.  Often this message is built into the Lambda handler framework itself through middleware components so that developers don’t have to do anything to get this happening.  In this way, we can compare the counts of **START** messages in the **notify-handler** service to the **COMPLETED** status messages in the **delivery-handler** service.

As you can see, for the No Flow Control test - the actual success rate was terrible.

![Actual success rate without flow control](/strands/observability/when-serverless-goes-wrong/no-flow-success.png)

Where the **Flow Control** test had a successful outcome.

![Actual success rate with flow control](/strands/observability/when-serverless-goes-wrong/flow-success.png)

## Conclusion

As you can see from this experiment, adding Flow Control between external communication points within each bounded context of your distributed system makes for a more successful event processing flow.  The other points that come to mind that are critical for building Event-Driven Architecture for success are:

* Understand your expected Volumes up-front and model them.  Understand what your real Transaction rate per second (TPS) is.
* identify the rate limits and constraints of systems you interface to.  It’s important to understand where the scale clashes will occur.
* Add flow control to your Bounded Domains to assist with scalability boundary clashes.
* Use structured logging to enable forensic analysis when things go wrong.
* Emit some simple messages to highlight the beginning of lambda execution and another to highlight whether it was successful.  Use these to create metrics for comparison and to measure the success of your system.
* Think about Business KPI metrics - I keep saying this but do think about them - they are also important, and the absence of a business metric can indicate something not working correctly. 

Observability is important, and it’s not that hard to get it right.  With some planning and the useful techniques highlighted here, you can quickly measure your serverless system’s success!


When Serverless Goes Wrong


I have been laser-focused on observability recently and thinking deeply about this topic.  Today, there are a ton (a literal ton!) of third-party observability tools.  Each one offers its take on resolving the observability problem.

The more I think about it, the more we need to stick to the basics and work with the tools provided by our cloud providers.  The reason for this is quite simple - serverless systems are designed to be hyper-scalable.  In an event driven system running at scale, data moves very quickly, and when something does go wrong, it happens so fast that by the time we notice, it has probably spiralled out of control, or the truth is buried in a mountain of log data.

Anytime something goes wrong, we will be looking back in time to work out what happened.  We will be forensically dissecting the data we have available to decide the root cause and the eventual resolution.  We need to understand the following:

- Why it went wrong?
- How it went wrong?
- What events or transactions went wrong, and how can we identify them for recovery?

In the cloud, systems must be designed to recover from failure, which can be manual or automatic; ideally, recovery will be automated.  To do this, the primary thing we need is enough data to observe what our system did leading up to the problem, during the problem and after it.

> We need the **ability** to **observe** what happened in the past.  That is the goal of systems being *Observable*.

# What makes Good Observability?
In the context of looking back when something has gone wrong, the number one thing you will need is great logging!  Without logs, you will never learn about what went wrong - so we need to be thinking about how we can make our logs work better:

- Capture meaningful messages about every phase of transaction processing
- Be able to identify which micro-service log data came from
- Identify the source code emitting the log entries (function, module, line)

> Our Logs must Tell a Story

I encourage teams I work with to think of their logs as “Telling a Story”.  The log entries should be meaningful about what the code is doing without being too verbose.  I also encourage emitting log entries for a Lambda function to indicate the Lambda lifecycle of when processing - Started, Completed or Failed.  This logging is often placed into middleware or framework code so that it happens constantly.  The less you need to rely on developers sticking to standards, the better! 

![Image showing a detailed log story](/strands/observability/getting-started/log-story.png)

> Logs must be structured

It is one thing to write logs that “Tell a Story”, but we need meta-data to be emitted with every log message, or we won’t be able to do enough searching, grouping and slicing and dicing of our logs when it matters.  To do this, our logs must be emitted in a structured JSON format which is recognisable by all log aggregation tools available today on the market - even Cloudwatch!

The image below shows how fields are extracted by Cloudwatch Insights, which turns your log properties into database table attributes that you can filter on, group by and sort.  This makes understanding how something went wrong a breeze and enables you to use `stats` functions to create new metrics!

![Image showing how structured logs look](/strands/observability/getting-started/structured-logs.png)

> Don’t just emit technical metrics. Think about business KPIs and design metrics around those.

Logging is not enough. Metrics are powerful and can be monitored to raise alerts to notify operational staff or developers that something needs to be looked at.  With metrics, we can apply Anomaly Detection [Using CloudWatch anomaly detection](https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Anomaly_Detection.html), which uses AI models to learn about your system’s behaviour and then alert when the behaviour changes.  This is useful because problems occur due to metric levels being too high, but the absence of a metrics is just as important! 

The third pillar of Observability is tracing - visualising a transaction’s path through your system is powerful, and tracing unlocks this capability.  Alongside visualisation, tracing also provides performance metrics so you can monitor the speed of your functions and different processing sub-sections of your transaction processing.

Combining all three pillars - Logs, Metrics and Traces, will provide you with Good Observability of your system when things go wrong.  

I have deliberately focused on achieving observability without using third-party tools, mainly because there are so many!  But also, each tool offers up different capabilities, so you need to be able to observe your system from the very start and allow time to choose a third-party observability tool if you feel this is required to meet your needs.

We need to remember when running an Event Driven system at scale, it will be the detail and the story your logs tell that will help you understand what happened.  So don’t get hung up on needing “immediate feedback” because by the time you react to an alert, seconds will have passed, and a few hundred or thousand transactions will have gone past already.  So be prepared to break out your forensic kit and uncover the story of the transactions that went wrong with nicely detailed logs.

This should be easy to do now!


Getting Started


> One of the AWS Leadership Principles is to “Learn and be Curious” - this encapsulates the **Observability Mindset** perfectly!

Observability is a deep and wide field with many tricks, tips and tools to try and achieve the nirvana of the “Perfect Observable System”.  Observability is not new, it is not limited to Serverless -  so why all the fuss on Observability and the proliferation of tools in the market to achieve perfect Serverless Observability?

The answer is simple - the industry was sold Serverless technology to achieve **Business Value** faster.  So the whole world went crazy and started building with AWS Lambda technology to do just this.  We built fast, lean, and stopped thinking about all the core principles of good development - after all, it was just a cloud function, it isn’t complex, and the code is simple - how hard can it be?

We forgot many things in getting to business value faster - I have written about this in an earlier blog post - [What I forgot at the beginning](https://blog.walmsles.io/serverless-testing-part-1-what-i-forgot-at-the-beginning).  This article is focused on Serverless Testing, but one of my main observations across the Serverless community at this time was the lack of focus on Software Architecture.  This is the main underlying theme of my whole testing series.  In a rush to Serverless and **faster business value**, we have also forgotten about good logging and a level-headed approach to thinking about how we understand what our distributed Serverless system is doing.

> We have forgotten about Observability thinking!

# What is the Observability Mindset?
[Observability](https://en.wikipedia.org/wiki/Observability) is the ability to infer knowledge about the internal states of a system based on its external outputs.  The Observability mindset is about thinking up-front about how you will ask questions about your system to work out what it is doing and whether it is behaving normally.  This means we need to think about how we are going to emit data and telemetry about our system functions so that people responsible for operating the system can have confidence it is working correctly.  Observability is a system design and development task that needs to be thought about up-front before we start building - because it is always harder to go back and put it in when you have a Serverless project being built quickly, in pieces by a large team.  

The mindset is also wholistic and not focused on just technical telemetry data - actual business KPIs must be designed and built-in so you can understand how customers use your system and whether you are building the right thing for your customers.  So its not just about - do we have bugs or errors but:

- Are your customers using the new feature
- Are our core processes running slower / faster after an update
- How long do customers spend using a new feature - lots, not a lot - what did you expect and want them to do?

# Why is Observability Important
Now that I read that heading back, a part of me is sad inside.  I had hoped that it was obvious why Observability is important, but unfortunately, it’s not that obvious after all.

Observability is important so we are able to:
- understand how people use our software
- understand whether our SaaS system is correctly priced so we are profitable
- Are our customers using our product the way we thought they would
- being able to reason about why an error occurred

These are some of the main reasons I came up with why Observability is important.  The last one is also an important reason for good logging, metrics and tracing.  With a Serverless system running at scale, by the time a monitoring alert goes off or an error is noticed - the error is likely minutes old and drowned in a sea of transactions.  So you must be able to interrogate your logs to work out what happened to a transaction, where it was processed and why it failed - all of this is impossible without good, structured logging with key fields to identify events passing through your system.  This is where frameworks like the [AWS Lambda Powertools](https://awslabs.github.io/aws-lambda-powertools-python/core/logger/) become important - this framework provides a structured logging utility covering multiple languages - Java, Python, Typescript and dotNet.  

![Example of Powertools Logger output](/strands/observability/mindset/logger.png)

The logging utility provides configuration to extract key fields from your event data to inject into every log message, which enables you to interrogate your logs using Cloudwatch Insight queries.

Next time you are starting a new project or picking up an old one - ask yourself these 3 simple questions:

1. Do I understand how my users use my system?
2. When something goes wrong (at scale), will I be able to find enough data to identify exactly what went wrong?
3. Do I know I am charging my customers correctly and can calculate my margin per customer?

If you can answer these three simple questions, you are on your way to having an Observability Mindset.  It doesn’t stop there - Observability changes and evolves the same way your software does - so don’t stand still, keep evolving, querying, observing and understanding.


Getting the Observability Mindset

# Watch for Scalability Boundaries

One of the biggest problems with deploying code and building services on a highly scalable platform is when you need to interact with an external service which may not be at quite the same scale.  This is what I call a **Scalability boundary**, a place where different scales meet and are likely to clash.

## The Importance of Flow Control
Serverless systems are high scale systems, and in my experience, high scale event driven architectures need flow control.  Flow control in enterprise architectures typically refer to SQS Queues or Kinesis Streams on AWS.  A place where transactions are persisted while they wait to be processed by one or more consumers.  The key is to tune the maximum concurrency on the Event Source Mapping Lambda trigger so that your system throughput at the scalability boundary point does not exceed the external integration endpoint’s rate limit.  The Event Source Mapping maximum concurrency setting is essential in ensuring Lambda execution throttles do not cause issues with events being lost or going straight to your Dead Letter Queues which can happen when you only use Lambda concurrency as a throttle control.   Where Event Source Mapping triggers are not available you will need to rely on Lambda reserved concurrency setting to control transaction throughput and clients will need to handle any throttles which occur. On AWS, if you are using Eventbridge, you can also leverage API Destinations, a purpose-built integration service with in-built controls for rate limiting.

> Know your Volumes!

## Understand your Volumes
This is why I keep saying - “Know your volumes”! Calculate your volumes and understand your system throughput. Putting together a volumetric model in an Excel spreadsheet will help you plan for performance testing and give you a reference point to the “whys” of the system architecture and service configurations.  Volumetric models are an important architectural tool many architects do not think about and ultimately end up with scalability boundary clashes.  Volumetric models also inform the future you about the "whys" involved in creating your solution today.

There are a few different ways scalability boundaries become problematic for Serverless systems.  The most common scalability boundary problems are listed below.

- Connection Oriented Services with finite connection limits.
- External API services that are rate limited.
- Slow external API services, essentially behaving like a rate-limited API.
- Not understanding how services work along with quotas and limits that apply.

> Understanding how the Services you use work is critical

## But it Worked in Testing?

I have had this experience when integrating into an external partner API secured using OAuth as the authentication mechanism.  In this instance, the API, which was poorly documented, had authentication session limits where each API key was limited to 10 tokens.  When the token number exceeded ten (10), they instantly invalidated the oldest token.  This all worked fine in development and testing, but as soon as the system operated at scale, there were problems and work was needed to change how we dealt with OAuth API tokens.

> If you are building an API, ensure you have good documentation!

## What happens when Scale Clashes?

When two systems clash at scale, the lower volume service will be overrun and become extremely slow or totally non-responsive, depending on how bad the scale difference is.  I have seen this often when a redis cache that is not properly sized.  When Redis gets overrun, what usually happens is the connection processing sub-system is super busy, and Redis spends most of its CPU trying to service connection requests resulting in low performance for actual cache queries and lookups.  With Redis running slow, your lambda code trying to use it will increase in execution duration due to wait times. Often, your lambda execution will timeout, and you will lose the lambda event unless you have designed your system for failure recovery.  When I witnessed this problem with Redis, it took some time to trace it to Redis since I saw timeouts and throttles elsewhere in my system, not particularly where Redis was involved, so you end up chasing your tail for a while before you get to the fact that the Redis cluster simply wasn’t right-sized!

> Lambda Duration is a Key metric to Monitor and anomaly detection across this metric is awesome!

When problems occur at your scale boundaries, lambda duration typically spikes up. I monitor lambda duration in cloud watch dashboards and activate anomaly detection on this metric.  **Duration** is a key metric to monitor since this will reflect when heightened latency affects your system, which generally means a scalability clash is occurring.  

> API latency, the time to execute an API call, is the biggest enemy of extreme-scale solutions like AWS Lambda. You need to understand how to control this aspect and monitor for problems.

## Resolving Scalability Clashes

In AWS, controlling scalability boundaries comes down to flow control.  This means using a buffered event mechanism such as an SQS Queue or a Kinesis stream.  These two managed services will create a persistence layer and store events until they can be processed and smooth out any spikes in your traffic.  

You control the flow of events by setting the **maximum concurrency** on the Event Source Mapping of your Lambda trigger which will ensure the maximum Lambda concurrency does not go too high.  Where an Event Source Mapping trigger is not available you can set a specific **reserved concurrency** value on your lambda functions which will cause the event trigger to throttle due to the lambda configuration denying the execution.  This means the trigger source will need to be looked at for specific behaviour controls when lambda throttles the invocations.

It comes down to knowing how your services work at scale and how you can take control of that and smooth out the transaction spikes when they occur at dangerous levels.  Learn about the services you are working with, run them at scale and understand how they fail.  It is the only way to build a fault-tolerant system which is the best kind to build in the cloud with Serverless services.


When Serverless Goes Wrong

Getting Started

Getting the Observability Mindset

Watch for Scalability Boundaries

Fundamentals

Observability Visuals