--- title: "Monitor and optimize S3 storage with Amazon S3 Storage Lens metrics" description: "Learn how to visualize S3 Storage Lens metrics in Datadog to spot cost-saving opportunities and understand data protection practices across your organization." author: "Ryan Warrier, David M. Lentz" date: 2021-12-09 tags: ["infrastructure monitoring", "aws", "amazon s3", "storage"] blog_type_id: the-monitor locale: en --- With Amazon S3's scalable object storage, you can store and manage billions of objects across multiple AWS accounts, regions, and storage classes. [S3 Storage Lens](https://aws.amazon.com/s3/storage-analytics-insights/) provides 29 useful metrics that give you deeper visibility into your S3 usage and activity across your entire organization. We are proud to be a [pre-integrated AWS partner](https://aws.amazon.com/blogs/storage/access-your-amazon-s3-storage-lens-metrics-in-aws-partner-applications/) using the new [CloudWatch publishing option](https://aws.amazon.com/about-aws/whats-new/2021/11/amazon-s3-storage-lens-metrics-cloudwatch/) to bring S3 Storage Lens metrics into Datadog for enhanced S3 storage monitoring. In this post, we'll show you how this integration can help you: - Monitor [S3 storage trends](#get-more-insight-into-s3-storage-trends-in-any-context) alongside your other infrastructure and applications - Optimize [S3 costs](#optimize-s3-costs-across-all-of-your-accounts) - Ensure that your organization is following [data protection](#ensure-that-your-s3-data-is-protected) best practices ![The S3 Storage Lens out-of-the-box dashboard shows an organization’s total storage, bucket count, object count, and storage breakdown.](https://web-assets.dd-static.net/42588/1776291364-amazon-s3-storage-lens-monitoring-datadog-s3-storage-lens-datadog-dashboard.png) ## Get more insight into S3 storage trends in any context Datadog's [out-of-the-box dashboard](https://app.datadoghq.com/dash/integration/30582/amazon-s3-storage-lens)—shown above—lets you easily visualize and explore all [S3 Storage Lens metrics](https://docs.datadoghq.com/integrations/amazon_s3_storage_lens.md), including the number of objects and bytes you've stored in S3 over time. Other widgets on the dashboard help you monitor your S3 cost efficiency, data protection practices, request activity, and errors. You can use the dashboard's [template variables](https://docs.datadoghq.com/dashboards/template_variables.md) to easily filter your metrics by AWS organization, account, region, bucket, and [storage class](https://aws.amazon.com/s3/storage-classes/). This granular view into your S3 Storage Lens metrics can help you understand organization-wide changes and trends in your S3 usage, costs, and security. And you can customize your dashboard to add context to your S3 Storage Lens metrics. Once you've [cloned](https://docs.datadoghq.com/dashboards.md#clone-dashboard) the dashboard, you can add widgets to track the performance of your infrastructure and other [AWS services](https://docs.datadoghq.com/integrations.md?q=amazon) alongside your S3 Storage Lens metrics. ## Optimize S3 costs across all of your accounts S3 Storage Lens metrics provide information about non-current object versions and delete markers, as shown in the screenshot below. These metrics represent opportunities to reduce your S3 storage costs by deleting unused objects. You'll also see information about incomplete multipart uploads, which can produce unusable data that remains in S3 until you explicitly delete it, for example by using an [S3 Lifecycle rule](https://aws.amazon.com/blogs/aws-cloud-financial-management/discovering-and-deleting-incomplete-multipart-uploads-to-lower-amazon-s3-costs/). ![The cost efficiency section of the S3 Storage Lens dashboard shows metrics describing versioned objects, delete markers, and incomplete multipart uploads.](https://web-assets.dd-static.net/42588/1776291368-amazon-s3-storage-lens-monitoring-datadog-s3-storage-lens-cost-efficiency.png) Datadog's S3 Storage Lens integration also helps you understand how your organization accesses data in S3, which can be useful for further optimizing costs. AWS charges for data transfer and retrieval per request and per byte, and the [costs](https://aws.amazon.com/s3/pricing/) vary across storage classes. Some retrieval methods (such as the [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html) SDK action) return an entire object, but you may be able to reduce costs by using [Amazon S3 Select](https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html) to query the object using SQL, allowing you to retrieve only the data you need. The screenshot below shows an excerpt of the out-of-the-box dashboard that can help you understand your organization's costs. For example, you can view a toplist of buckets that have the lowest data retrieval rates (indicating that they store infrequently used data), and consider moving them to a lower-cost storage class. You can also see the number of bytes scanned by S3 Select—indicating the amount of data processed by queries—and the number of bytes returned—indicating the aggregate size of all result sets. The buckets with the most bytes scanned might represent an opportunity to improve the [performance](https://docs.aws.amazon.com/AmazonS3/latest/userguide/selecting-content-from-objects.html#selecting-content-from-objects-using-byte-range) and reduce the [cost](https://aws.amazon.com/s3/pricing/) of your S3 Select queries by using the [`ScanRange`](https://docs.aws.amazon.com/AmazonS3/latest/API/API_ScanRange.html) parameter to scan only a subset of an object's data. ![The S3 Storage Lens dashboard shows bytes downloaded and the percentage of data retrieved, bytes scanned and retrieved by S3 Select queries.](https://web-assets.dd-static.net/42588/1776291373-amazon-s3-storage-lens-monitoring-datadog-s3-storage-lens-retrieval-rate-scanned-bytes.png) ## Ensure that your S3 data is protected AWS [best practices](https://aws.amazon.com/blogs/security/top-10-security-best-practices-for-securing-data-in-amazon-s3/) prescribe a number of steps you should take to protect your S3 data against risks, including loss, corruption, and unauthorized changes. S3 [guarantees durability](https://docs.aws.amazon.com/AmazonS3/latest/userguide/DataDurability.html) to guard against data loss, but you still need to configure each bucket properly to take advantage of S3's features for [replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html), [versioning](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Versioning.html), and [object locks](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-lock.html). The S3 Storage Lens dashboard provides a quick summary of the percentage of total storage that is encrypted, replicated, and protected with an object lock, as shown in the screenshot below. You can also monitor these metrics over time and even create [alerts](https://docs.datadoghq.com/monitors.md) to notify you automatically if your data protection metrics drop below a threshold you specify, allowing you to quickly identify and remediate any incorrectly configured buckets or objects. For example, if you see a steady decrease in the percentage of objects with an object lock enabled, it could indicate that an application became misconfigured and began to create unprotected S3 objects. The toplists at the bottom show the buckets with the greatest risk of data loss, corruption, and exposure, which you can use to prioritize improvements to your S3 data protection. ![The data protection section of the S3 Storage Lens dashboard shows the percentage of S3 data that is encrypted, replicated, and protected with an object lock.](https://web-assets.dd-static.net/42588/1776291378-amazon-s3-storage-lens-monitoring-datadog-s3-storage-lens-data-protection.png) And for continuous visibility into the protection and security of your S3 data, you can use Datadog Cloud SIEM's [out-of-the-box rules](https://docs.datadoghq.com/security_platform/default_rules.md) to monitor S3 permissions automatically. ## Get started with S3 Storage Lens monitoring in Datadog S3 Storage Lens complements our existing [S3 integration](https://docs.datadoghq.com/integrations/amazon_s3.md) to provide even deeper visibility into S3. To get started, configure S3 Storage Lens to [publish metrics to CloudWatch](https://docs.aws.amazon.com/AmazonS3/latest/userguide/storage-lens-cloudwatch-enable-publish-option.html), then [enable the S3 Storage Lens integration](https://docs.datadoghq.com/integrations/amazon_s3_storage_lens.md). If you're not already using Datadog, start today with a .