AWSvAzure - Object Storage - Valey Key Pattern Part 2

See my post on Keep Secure.

https://www.keepsecure.ca/blog/awsvazure-object-storage-valet-key-pattern-part-2/

AWSvAzure - Object Storage - Valet Key Pattern Part 1

So far in this series, we've covered a lot of the technical aspects of object storage and have explored how they are implemented in both cloud providers. For this post, I'd like to examine a cloud architecture pattern called the valet key pattern, and discuss how to implement this in both Azure and AWS. In this post, lets talk about what the valet key pattern is.

Please note that most of this content is based off of the cloud architecture patterns book. https://docs.microsoft.com/en-us/azure/architecture/patterns/valet-key

Traditionally, web applications have always acted as a gateway for user access to protected resources. This becomes quite apparent when you think about databases that back these applications. They are a protected resource, and the web application provides limited access for users to interact with those database resources.

Databases are not the only resources that web applications can act as a gateway for. For many applications, other services such as storage could be in use. Think about an application that ingests video for downstream processing/hosting. Other examples could be APIs that place transactions to back-end systems.

On-premises systems architecture would general hide these types of protected resources from external access. This is obviously a security best practice, especially when the same "storage" engine is being used for all types of data. In the cloud, however, it is possible to create resources for only a specific purpose. Further to this, cloud resources such as storage and queues are already available on the internet by default. The protection mechanism of those resources have shifted from network controls to user based controls.

Because of this, both internal facing and external facing web applications can make use of the valet key pattern. The pattern looks something like this:

 

Effectively, instead of the web application proxying access to protected resources, it acts as a gatekeeper for credentials/access decision only. Let's take the example of a user wanting to upload an image for processing purposes.

1. User requests access to upload a file for processing
2. Application checks the request (authentication and authorization)
3. Application provides a temporary access token that grants the user appropriate permissions to perform the requested action
4. User interacts with the protected service to perform the desired action

There are obviously pros and cons to the above approach, which I won't dive in to here. Most of them are trying to strike a balance between usability of the solution and bandwidth/performance considerations.

In terms of object storage, this pattern is applicable in both AWS and Azure. For AWS, we will be using the secure token service to provide a temporary token and we will take a look at signed URLs. For Azure, we will simply provision a shared access signature with the required permissions.

 

AWSvAzure - Object Storage - Monitoring

Monitoring of object storage services in the public cloud is tightly integrated with the general monitoring services (read AWS CloudWatch, Azure Monitor) of those cloud providers. The goal for this post is to discuss how monitoring is conducted.

	AWS	Azure
Metric Types	Operation counts HTTP Response Counts Bucket Size Bucket Object Count Latency Ingress/Egress	Operation Counts HTTP Response Codes Account Size Account Container Counts Account Object Counts Latency Ingress/Egress Availability Specific Error Counts
View	CloudWatch Metrics S3 Dashboard	Azure Monitor Azure Tables (via Storage Explorer)
Metric Intervals	1-minute	Hourly / Minute
Metrics Storage	CloudWatch Metrics	Azure Tables
Retention	CloudWatch Defaults	Configurable
Billed	Yes (By Metric Captured)	Yes (By data retained / transactions)
Default	Bucket Size Bucket Object Count	Hourly Metrics Account Container Counts Account Object Counts Account size

 

From a metrics types perspective, both AWS and Azure capture similar types of information. The only real difference is the amount of metadata that is captured by Azure about a particular request. This allows you to monitor specific conditions (for example, SAS token failure) and respond/alert appropriately.

For more information on the metrics available in AWS, please see https://docs.aws.amazon.com/AmazonS3/latest/dev/cloudwatch-monitoring.html.

For more information on the metrics available in Azure, please see https://docs.microsoft.com/en-us/rest/api/storageservices/storage-analytics-metrics-table-schema

Here is an example snippit from the Azure table for the storage account I've been using:

 

 You can also view this data via Azure Monitor, and use the built-in tools to hack/slash as you see fit.

 In AWS, the default view is cloudwatch. As far as I know, there is no direct access to the underlying data.

 

The only other real difference to point out is the retention period. Since all storage account data is logged to Azure Tables, you can customize the service to retain only the amount of logs you require. In CloudWatch, the defaults are derived based on the interval the data is captured. The price for that storage is included when you turn on the metric.