All the logs and event data collected from external network and AWS cloud infrastructure can be forwarded to AWS Kinesis Stream in order to derive real-time insights such as generate metrics, live dashboards, correlate event data to identify threats and much more.
In order to derive such insights, these data streams must be sent to a machine which forwards the data to a DNIF Adapter(AD). A docker image namely AWS Log Forwarder is available to tap into the Kinesis Data Streams and forward logs from the required shard-id to AD.
Steps to setup an AWS Log Forwarder
Step 1: Prerequisites
Before moving onto the steps for installation, make sure the following requisites are met:
- Admin Access to your AWS Console.
- Access to the below mentioned websites: https://www.docker.com/community-edition https://hub.docker.com/r/dnif/aws-forwarder/
- Hardware prerequisites:
Hardware requirements depends upon the Events Per Second(EPS) of logs expected. In high EPS scenarios, a reckoner will be provided by [email protected] separately on request.
For general use, you can start with 2 Core CPU, 8GB RAM and 150GB HDD instance.
- OS prerequisites:
We recommend that you install the latest 64-bit version of Ubuntu or CentOS.
- Network prerequisites:
Make sure port 514/UDP is open from aws-forwarder to Adapter(AD). Make sure the domain www.amazon.com is accessible or whitelisted for aws-forwarder.
Step 2: Docker Installation
Before you install DNIF-AWS-log-forwarder image, you need to first install Docker. There are two variants of Docker: Docker-CE (free version) and Docker-EE (paid version).
Note: The OS architecture should be 64-bit and not 32-bit, as the latter isn’t supported by DNIF services.
See Docker installation guidelines in order to check which version of Docker your OS supports.
2.1.0 Testing Docker Daemon
Verify that Docker CE is installed correctly by running the hello-world image.
$ sudo docker run hello-world
This command downloads a test image and runs it on a container. On successful execution, it prints an informational message and exits.
2.2.0 Download the configuration file
Download the [configuration file](https://dnif.it/artifacts/docker-compose.yml) that comprises of all the configuration information: Download link.
2.3.0 Create a separate folder for the configuration file
Create a separate folder at a location of your choice and name it aws-forwarder.
Note: This folder will store the configuration file that needs to be accessed every single time you run aws-forwarder services. Ensure that the folder is easily available and is not deleted.
Step 3: Move the Configuration File
- Once you’ve downloaded the file as per Step 1, verify its name is docker-compose.yml.
- Move the downloaded file into aws-forwarder folder that was created in Step 2.4.0.
Step 4: Edit the Configuration File
- Navigate to aws-forwarder.
- You’ll now be editing the file docker-compose.yml by replacing the highlighted fields with their respective values. Fields to be edited are highlighted below:
- /var/tmp/- Replace this text with the path where you would like to keep service logs of DNIF forwarder.
- us-east-1- Replace this with the region code of your AWS instances.
- AKXX…XX6A- Replace this with your Access-Key-ID.
- AbcXXXX…XXeF- Replace this text with your Secret-Access-Key.
- 192.168.01.XX- Replace this IP Address with the Public/Private IP of DNIF-Adapter that is in connectivity with AWS-Log-Forwarder host-machine.
- KatrinaStreaming- This is a user defined tag name for streaming.
- shardId-000000000000- You can define a specific shard-Id from Kinesis Stream for log forwarding by changing its ID here, else leave as it is (000000000000).
- awslf.log- This is a default name for AWS-Log-Forwarder service log. You can change its file name by editing this. Remember, not to change /dnif/ directory path.
- You can also change service logging level by changing this value. Supported log levels are mentioned below:
In case you are finding it difficult to get your Access-Key-ID and Secret-Access-Key. Please follow the steps as follows:
- Open the AWS Console.
- Click on your username on the top right and select Security Credentials.
- Click on Users on the sidebar.
- Click on your username.
- Click on the Security Credentials tab.
- Click Create Access Key.
- Click Show User Security Credentials.
Step 5 : Executing the Configuration File
- For the final step, let’s first open the Terminal window.
- Now navigate to the path where you have created the aws-forwarder folder [From Step-1 and 2].
Next, execute the configuration file using the following command:
$sudo docker-compose up -d
The time taken by the above command to fetch the DNIF image from the online repository will depend on the speed of your internet connection. This installation is one-time process to fetch image. From second time onwards, the service starts directly.
Run the below mentioned command from same aws-forwarder folder to stop service.
$sudo docker-compose down
The above steps help to capture relevant log data from multiple services within AWS. Once all the services are configured successfully, you can further derive some interesting insights from those data by reading our help document on Analytics for AWS.