Using EventStore


EventStore is a feature available on DNIF to manually upload logs into a custom index using standard file formats like json, csv, xls and xlsx, thus enabling you to do retrospective analysis on this data. You can do a search, make reports, visualize data on a dashboard and so much, much more cool stuff!

To create an EventStore

Login to DNIF console and setup a connection to the required repository from the “Connections” option that is available in “MANAGEMENT” tab.

Once you are connected, navigate to the “SEARCH” tab and click on the “Settings” icon, which is a gear-like icon at the top left corner of the screen.

image

Now, click on “Add Event Store” from the “Options” menu. The following window to create an Event Store will appear.

image

Here, give your store a name in the “Store Name” column. This name must be in small character-set with no numbers or special characters.

Next, you have to select any one of the “Store Type”s:

  • Create an empty store
  • Create and upload
  • Read from file

Create an empty store

This option allows you to create an empty storage that can be used later to populate data from HTTP/HTTPS API in simple json format as input.

To forward json logs to API please refer help document Forwarding to API

Create and upload

This option allows you to create a new storage and uploads the log data file at once.

image

As soon as the store is ready to use, the “Create and upload” tab appears with two additional options to “Upload” the file and “Notify me by email”, along with the “Store Name” and “Store Type” options.

Notify by email

If this option is checked, a user will receive a notification email when the uploaded file gets indexed in the created store. You need to complete the SMTP configuration in order to use this feature.

Upload

If you want to upload a file to add into the Event Store, this is where you can do it. Clicking “Upload” allows you to browse a log from your machine in any of the supported formats - csv, json, xls or xlsx. This uploaded file must be less than 100 MB in size. Once uploaded, it will get queued for indexing in the newly added store.

Note - Data-type in the supported file format is described later in this page.

To Read from File

This option allows you to upload files greater than 100MB which shall appear as a list in the drop down menu called “File List”. The name of the file shall be visible in the drop down menu once the file has been moved to a folder named UPLOADS within the installation directory.

image

In the “Read from file” tab, you will find some options, such as:

  • “Store Name” to name your storage,
  • “Store Type” to select the type of upload,
  • “Notify me by email” to receive a notification email when the uploaded file gets indexed in created store.

You also get an additional option “File List”. Once the log file is uploaded on the DS or A10 UPLOAD folder, you will get the uploaded file name listed in the “File List” option. Next, select a filename and click on the “Save” button.

EventStore Supported file types and Data-format

Event Store supports the following file formats to upload logs:

  • CSV
  • JSON
  • XLS/XLSX

CSV

To upload logs in comma separated values .csv, keep the following points in check:

  • The first line of the csv file must contain comma separated field-names.
  • Values for fields need to be in the same order of the first line in a file.

    Example:

      LogType,LogEvent,EvtLen,EventID,HTTPMethod,SrcIP,SrcCN,SystemTstamp,TXLen,GivenName
    WEBSERVER,"<181>Sep  5 16:48:05  APACHE-SYSLOG[13054]: 111.222.111.222- - [05/Sep/2017:16:48:05 +0530] ""GET / HTTP/1.1"" 200 59426 ""-"" ""Mozilla/5.0 (X11; U; Linux i686; cs; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7""
    ",211,200,GET,111.222.111.222,IN,05/09/17 16:48,59426,Website
    WEBSERVER,"<181>Sep  5 16:48:05  APACHE-SYSLOG[13054]: 111.222.111.223 - - [05/Sep/2017:16:48:05 +0530] ""GET / HTTP/1.1"" 200 59426 ""-"" ""Mozilla/5.0 (X11; U; Linux i686; cs; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7""",210,200,GET,111.222.111.223,IN,05/09/17 16:48,59426,Website
    WEBSERVER,"<181>Sep  5 16:47:06  APACHE-SYSLOG[13054]: 111.222.111.224 - - [05/Sep/2017:16:47:06 +0530] ""GET / HTTP/1.1"" 200 59426 ""-"" ""Mozilla/5.0 (X11; U; Linux i686; cs; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7""
    ",211,200,GET,111.222.111.224,IN,05/09/17 16:47,59426,Website
    WEBSERVER,"<181>Sep  5 16:47:06  APACHE-SYSLOG[13054]: 111.222.111.225 - - [05/Sep/2017:16:47:06 +0530] ""GET / HTTP/1.1"" 200 59426 ""-"" ""Mozilla/5.0 (X11; U; Linux i686; cs; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7""",210,200,GET,111.222.111.225,IN,05/09/17 16:47,59426,Website
    
    

JSON

The Event Store supports a simple json file format with key:value pairs.

Note - We suggest you avoid using the nested json format, as they can cause issues with indexing.

Example:


  [
    {
      "LogType": "WEBSERVER",
      "LogEvent": "<181>Sep  5 16:48:05  APACHE-SYSLOG[13054]: 111.222.111.222- - [05/Sep/2017:16:48:05 +0530] \"GET / HTTP/1.1\" 200 59426 \"-\" \"Mozilla/5.0 (X11; U; Linux i686; cs; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7",
      "EvtLen": 211,
      "EventID": 200,
      "HTTPMethod": "GET",
      "SrcIP": "111.222.111.222",
      "SrcCN": "IN",
      "SystemTstamp": "05/09/17 16:48",
      "TXLen": 59426,
      "GivenName": "Website"
    },
    {
      "LogType": "WEBSERVER",
      "LogEvent": "<181>Sep  5 16:48:05  APACHE-SYSLOG[13054]: 111.222.111.223 - - [05/Sep/2017:16:48:05 +0530] \"GET / HTTP/1.1\" 200 59426 \"-\" \"Mozilla/5.0 (X11; U; Linux i686; cs; rv:1.8.0.7) Gecko/20060909 Firefox/1.5.0.7",
      "EvtLen": 210,
      "EventID": 200,
      "HTTPMethod": "GET",
      "SrcIP": "111.222.111.223",
      "SrcCN": "IN",
      "SystemTstamp": "05/09/17 16:48",
      "TXLen": 59426,
      "GivenName": "Website"


XLS / XLSX

Event Store also supports standard Excel file formats in both xls and xlsx type.

It is necessary to have field information in the very first line of the file that you are uploading.

Example :

image