I chose this project not because it's fancy, but because it's easy to understand, and shows a simple, efficient solution. By the way, I made this page. I have some frontend skills, too.
Background
In 2017, as one of my first projects at Amazon, I was tasked to design an API and implement a solution to enable customers to see how many AWS resources (EC2 instances, queues, lambda functions, S3 buckets, etc) they had in their account. My team, AWS Config, was in the configuration management space, so our focus was to observe a customer's AWS resources and alert them when configurations violate customer-selected rules. We would alert them to security and privacy-related misconfigurations, help them avoid high costs, and prevent shadow IT.
The main requirement of this project was to build a public-facing API that, when queried, would return the resource counts for each resource type in the account. As an example, the API might return that you have 2 IAM roles, 5 EC2 instances, 3 security groups, and 1 load balancer. The information from this API was also to be presented above the fold on the customer's dashboard. It had to be accurate and low-latency (<= 100ms), and stay as such even for a P100 customer such as Netflix, which could have tens of thousands of resources at any given moment.
Research
To guarantee low-latency, the data source should respond quickly to queries, regardless of the number of resources.
In AWS, relational databases are not allowed, unless you have a very specific need that must be justified to higher, org-level leadership. Even though it was somewhat off the table, I employed it in my research to test latency and to be able to query on my own to get the scope of our largest customers. We had a DynamoDB table that already had resources and their metadata (we'll call it customer-resources), so I dumped that table to an temporary RDS (AWS managed relational DB system) MySQL database and also a local database to analyze the problem and test query speed. My experiments confirmed my suspicion that the latency to query a relational database was going to be too slow.
A common solution in AWS is DynamoDB, a highly scalable key-value data store, but it lacked aggregation capabilities, meaning that any counting of rows or values was impossible. We had a table with the information I required, but my access pattern was unsupported.
Design
The final design was built around a new DynamoDB table (discovered-resource-counts) that would store resource types (keyed to an account) and their counts, kept up-to-date by two separate queues/workers. One worker would provide a snapshot of counts for each resource and replace the records in the table for an account. Another worker would increment or decrement counts based on the detection of new resource creation and deletion.
With this design, the GetDiscoveredResourceCounts API reads the precomputed counts directly from the table, providing constant-time responses, regardless of table or resource count size.
Initial Data/Anti-Entropy
It all starts with the complete resource count snapshot. For this I utilized an existing anti-entropy system 1 that, every 6 hours, would make API calls to all supported resource APIs to gather resources for each customer. This information then populates the customer-resources DynamoDB table. I added code to this process that counts resources as the APIs return them, and once all the resource counts are tabulated, sends that as a moment-in-time snapshot payload to an SNS topic (AWS pub-sub). I have a queue subscribed to that topic, and workers read the queue 2, using the information to update the discovered-resource-counts table, replacing any existing records for that account. This set of records is the baseline that the other process will use the keep the counts up to date.
Updating Counts
When a customer creates or deletes a resource (such as terminating an EC2 instance), a notification from the customer account is read by another AWS Config event handler system 3 that records that resource's state in the main customer-resources table. I added an event listener (Kinesis stream) on this table that streams changes to the table. That event stream is read by a service 4 with the responsibility of keeping track of table changes and checkpointing progress. It filters out any messages unrelated to addition or removal of records. It then feeds the create/delete updates to a queue, and the queue worker 5 updates the discovered-resource-counts table, incrementing the resource count in the table for "record added" events, and decrementing (and removes if now 0) for "record deleted" events.
Asynchronous Processing
Queues are part of the design to avoid tight coupling and allow systems to scale independently. The SNS topics themselves are not a requirement when using a queue, but they allow for fanout later to additional queues if the data is needed for other purposes, without affecting the operation of existing queues. They are cheap to add and a common pattern when using queues in AWS teams.
API
AWS documentation available here.
Request
The API GET request accepts:
- limit (max 100)
- nextToken - pagination token
- limit optional array of resource types: returns all types when not specified.
{
"limit": number,
"nextToken": "string",
"resourceTypes": [ "string" ] # optional
}Response
- nextToken - pagination token
- resourceCounts - list
- count
- resourceType (example: AWS::CodeDeploy::DeploymentGroup)
- totalDiscoveredResources the total number of resources recorded for your account (or total for requested resources)
{
"nextToken": "string",
"resourceCounts": [
{
"count": number,
"resourceType": "string"
}
],
"totalDiscoveredResources": number
}In Hindsight
Looking back, I could have investigated into hooking into the event system directly and avoided the event stream from the DynamoDB table. I don't remember the event system's architecture at the time, or if I investigated during my research. But if it had been plausible, I would have added code to it to send create/delete update messages to the update topic directly.
In later years they moved resource information to ElasticSearch (now OpenSearch), which was a nascent offering inside AWS, for the purposes of allowing customers to directly query their resources. The system above was likely replaced with queries to that system.