November 07, 2017 / by jburks / In guides /

Data Analytics with AWS S3 Logs

What is AWS S3?

AWS Simple Storage Service (S3) is storage for the Internet, designed to make web-scale computing easier for developers.

AWS S3 is a simple web services interface that you can use to store and retrieve any amount of data, at any time, from anywhere on the web.

Why S3 Logging is important?

An AWS S3 bucket can be configured to create ‘access log records’ which contains details on an access request made, such as

  • Request type
  • Resource worked with
  • Requester
  • Turn-around time
  • Processing time and date

Server access logs give bucket owners an insight into the nature of requests made by users not under their control.

  • Customized Dashboards for real time traffic flows.

  • Easy to implement use cases, as per security, business, and user requirements

  • Perform AWS S3 log analysis

  • Validate Usage

  • Monitor all data that resides within your AWS S3 bucket

  • Index, search and analyze performance and access logs

  • Generate reports and determine AWS usage patterns

By creating a single dashboard, system analysts or required stakeholders can have a comprehensive view of their storage platform. The dashboard can have multiple widgets, depending upon business or security use cases.

Shared below is the dashboard view, with some useful widgets, that you can create easily within DNIF:

Create widgets to monitor acitivities related to AWS S3 buckets

Widgets created for the dashboard above include : - Source based heatmap - Request monitoring - Error monitoring - Frequent visitor list - Total event count

These widgets benefit immensely while performing data analytics for effective threat hunting.

Use Cases

Source Based Heatmap

Create world wide heat map of source traffic

Geo tracking helps identify good traffic patterns and adjusts your AWS resources, as required, to serve particular regions better. S3 logs can provide actionable insights on where, when, and for how long your users were active in the last 24 hours. This information can prove helpful for cyber security professionals to perform possible data analytics.

Request Monitoring

Monitor web request

Here, we’re monitoring the user activity over the last 24 hours, by keeping track of the HTTP request methods and the count for these method calls.

Error Monitoring

Monitor AWS S3 errors

A wealth of critical information about your network and applications resides in the errors monitored and logged in AWS S3 bucket over the last 24 hours.

You can take a look at the complete list of Error Codes.

Frequent visitor list

List of all the Source IP Addresses that tried accessing a specific bucket in the last 24 hours, and their frequency.

Total Event Count

Count of all AWS S3 bucket logs for the last 24 hours.