Tag: How to Configure AWS S3 Inventory

How to Configure AWS S3 Inventory Reports

S3 inventory overview

• Flat file list of your objects and metadata, which is a scheduled alternative to the Amazon S3 synchronous List API operation.
• S3 Inventory provides our buckets or objects sharing a similar prefix, with the following output files for listing objects and metadata (daily or weekly).
• Comma-separated values (CSV)
• Apache optimized row columnar (ORC)
• Apache Parquet (Parquet)

Running S3 Inventory report is an important step for many compliancy requirements although efficient cost visibility and cost reduction strategy can be quite challenging for most organizations. If you want to reduce your S3 spending start with reviewing your data transfer costs:

1. Use CloudFront,
2. Redesign object locations,
3. Apply managed lifecycle for all S3 Objects,
4. Review and cleaning S3 Objects that are never accessed

Alternatively, you can get instant visibility on all of the above with our AWS Savings Report (Hey! It’s free). The report will instantly give you the visibility on the cost of your S3 service and help you to reduce the overhead/wasteful spending significantly.

For Inventory configuration follow the steps below:

The first initiated report may take up to a total of 48 hours.
1. Login to the Management Console and go directly to the S3 console from the following link https://console.aws.amazon.com/s3/.
2. From the Bucket name list, select the bucket that you wish to configure its S3 inventory.
3. Select the Management tab, and click on Inventory.

4. Select the Add new option

5. Fill in a specific name for inventory and start setting it up through those steps:

1. You can have the option to add a prefix for the filter to objects found in inventory who have names beginning with identical strings.
2. Select which destination bucket you would like the reports to get saved in. It has to be in the exact Region as that of the bucket whose inventory is being set up. It could be found in another account.
3. You can have the option to select a prefix for your destination bucket.
4. You have the choice to select how often to generate your chosen inventory.

For the Advanced settings section, you have the option to set the below:

 

• Select one of the following: ORC, CSV or Parquet output file format for inventory.
• Select the option Include all versions in the Object versions list to get all versions. (Default: only the current versions are included)

• In the Optional fields, you can choose any of the below to have included with inventory report:

 

• The Size: in bytes.
• The Last modified date: the latest date when either created or last modified.
• The Storage class: class where object is stored.
• The ETag: hash of your object. Shows alterations made to contents of an object solely, not the metadata. It could be an MD5 digest of object data, depending on the way this object was created and encrypted.
• The Multipart upload: Shows an object was uploaded as a multipart upload.
• The Replication status: Replication status of an object.
• The Encryption status: Server-side encryption which encrypted an object.

The Object lock configurations: Object lock status, which has the bellow settings:

• “Retention mode”: Level of protection of an object (Governance or Compliance)
• “Retain until date”: Date when locked object can no longer be deleted.
• “Legal hold status”: A locked object’s legal hold status.

 

• In Encryption section, select the server-side encryption option you want for encrypting your inventory report, or simply select None:

 

1. 1. • “None”: No encryption for inventory report.
      • “AES-256”: Encryption through server-side encryption of S3-managed keys (SSE-S3). (Works with a 256-bit Advanced Encryption Standard (AES-256))
      • “AWS-KMS”: Encryption through server-side encryption of Key Management Service (KMS) customer master keys (CMKs).

For encrypting inventory list file with SSE-KMS, give S3 permission for using KMS CMK.

1. Click on Save button.

 

What does S3 Inventory Consist of?

An inventory list file has:

-List of objects found in the source bucket

-Metadata for every object

Inventory lists are stored inside the destination bucket:

-As a CSV file (compressed with GZIP)

-As an Apache optimized row columnar (ORC) file (compressed with ZLIB)

-As an Apache Parquet (Parquet) file (compressed with Snappy)

Inventory list lists objects of S3 bucket with the bellow metadata for every single object listed:

• The Bucket name: Name of bucket that this inventory is for.
• The Key name: Unique object key name for identifying an object in the bucket. CSV file format gives a key name which is URL-encoded that should be decoded before getting used.
• The Version ID: The object’s version ID number, upon enabling versioning on this bucket.
• The IsLatest: Its value is Truefor current version objects.
• The Size: Size (in bytes).
• The Last modified date: Object creation date or the last modified date, whichever is the latest.
• The ETag: The entity tag is a hash of the object, reflecting alterations merely made to contents of the object (not to metadata). It could be an MD5 digest of the object data. Being so, relies on the way of creation of the object and encryption.
• The Storage class: Where an object is stored.
• The Intelligent-Tiering access tier: Access tier which may be frequent or infrequent for objects stored in the Intelligent-Tiering.
• The Multipart upload flag: Its value is Truefor objects uploaded with a multipart upload.
• The Delete marker: Its value is True for delete marker objects. (directly found in your report if it’s configured to include every version).
• The Replication status: Its value can be “PENDING”, “COMPLETED”, “FAILED”, and “REPLICA”.
• The Encryption status: Its value can be “SSE-S3”, “SSE-C”, “SSE-KMS”, and “NOT-SSE”. Objects that are server-side encrypted have an “SSE-S3”, “SSE-KMS”, and “SSE” that comes with customer-provided keys, “SSE-C”. Having a NOT-SSEis for objects that have no server-side encryption.
• The Object lock Retain until date: Locked objects may not be deleted until this date.
• The Object lock Mode: Has a value of “Governance” or “Compliance” for locked objects.
• The Object lock Legal hold status: Has a value of  “On” for legally held objects, or simply set “off” if not.

It is advised that you get a lifecycle policy created for deleting the old inventory lists.

AWS S3 Inventory provides you a combined view of your S3 buckets and helps your compliance needs, to calculate the cost of each bucket you can use AWS cost and usage reports or our S3 Cost calculator to forecast any planned changes.

uploading files to S3 bucket