_store


The _store directive is used to write and store filtered output, either temporarily in memory [in_mem] or permanently on Hard Disk storage [in_disk] for later use.

Syntax:

_store [in_mem | in_disk] <store_name> [stack_replace | stack_append | key_replace]

The _store directive uses the following options to store query output:

Filenode type:

  • in_mem: Stores output stack in memory i.e. temporary storage
  • in_disk: Stores output stack on hard-disk drive i.e. permanent storage

Stack writing instruction:

  • stack_replace: to replace the complete storage-stack or to create a new storage-stack
  • stack_append: to append fetched values to a pre-existing storage-stack
  • key_replace: to replace all values of pre-existing key/column in a storage-stack with new values

Filenode type:

  • in_mem:

    Using the in_mem option, you can write the result set for a query to a temporary filenode in the memory of the DNIF-container. This storage-stack is only available till the container or the host-machine restarts.

Example:

_fetch * from event where $Duration=12h AND $GivenName=App group count_unique $Action limit 100 >>_store in_mem action_list stack_replace

The above query will fetch distinct $Action from the machine whose $GivenName is ‘App’, and its result set will be pipelined to the directive, _store, to store the result in_mem with the store-name as ‘action_list’.

To retrieve or view the data stored in a storage-stack, please refer the _retrieve - Read from Filenode document.

  • in_disk:

    Using the in_disk option, you can write a result set on a permanent filenode on the DNIF-container’s Hard Disk Drive [HDD]. This storage-stack will remain available unconditionally to the container or host-machine until restart

    Example:

    _fetch * from event where $Duration=12h AND $GivenName=App group count_unique $Action limit 100
    >>_store in_disk action_list stack_replace
    

    The above query will fetch distinct $Action from the machine whose $GivenName is ‘App’, and its result set will be pipelined to the directive, _store, to store the result in_disk, the DNIF HDD, with the store-name as ‘action_list’.

    To retrieve or view the data stored in a storage-stack please refer the _retrieve - Read from Filenode document.

Stack Writing Instruction:

  • stack_replace:

    This writing instruction is commonly used while creating a new store, either in memory or on disk. Running the stack_replace instruction on pre-existing stack will truncate and overwrite it with the latest output.

    To further understand this concept, let’s create a fresh in_disk stack:

    _fetch * from event where $Duration=1h AND $LogType=WEBSERVER group count_unique $SrcIP limit 10
    >> _store in_disk src_list_web stack_replace
    
Screenshot of the entries stored

To check entries that have been stored, let’s retrieve the stored stack, src_list_web, from the previous query:

_retrieve src_list_web

Screenshot of the entries stored

As Shown above, we’ve successfully created a storage-stack, src_list_web, that contains a list of distinct source IPs, $SrcIP, that had connected to our web server in the last 1 hour, with their associated event counts in the count_unique column.

  • key_replace:

    The key_replace writing instruction is useful when there is a need to update the data stored with the latest search values for certain pre-existing keys. This instruction will automatically append a new key to the store.

    Using the above example, let’s execute the following query on the store, src_list_web:

    _fetch * from event where $Duration=5h AND $LogType=WEBSERVER group count_unique $SrcIP limit 5
    >> _store in_disk src_list_web key_replace $SrcIP
    

    If you’ve noticed, we’ve added $SrcIP in the end of the _store query. Here, the field $SrcIP is a key for which we want the updated results to be fetched from the _fetch query.

Screenshot of the result set

Let’s now retrieve the updated store, src_list_web, from the previous query.

Screenshot of the entries retrieved

As you can see from the highlights above, as the result for _retrive, the value of count_unique for the three IPs were updated, as they were already in the store. Besides, two more entries were added at the bottom of the list.

So, in a nutshell, key_replace will update the relevant values for a given key if it is already present in the stack. In case of a new entry, key_replace will append the key-value pair.

  • stack_append:

    Unlike key_replace, this instruction can only append the fetched result at the bottom of the stack

    Continuing with the same example of the src_list_web store getting updated by the key_replace query, here’s how stack_append would work:

    _fetch * from event where $Duration=3h AND $LogType=WEBSERVER group count_unique $SrcIP limit 2
    >> _store in_disk src_list_web stack_append
    

    In the query above, we are retrieving the list of active IP addresses from our web server for the last 3 hours, and are adding them to the existing store, src_list_web, using stack_append.

Screenshot of the result set

Now, let’s _retrieve the store src_list_web:

Screenshot of the entries retrieved

In the above screenshot, highlighted in red, we see the newly added results at the bottom of the stack, whereas similar IP addresses were already available in the store.