The _store directive is used to write and store filtered output, either temporarily in memory [in_mem] or permanently on Hard Disk storage [in_disk] for later use.
_store [in_mem | in_disk] <store_name> [stack_replace | stack_append | key_replace]
The _store directive uses the following options to store query output:
- in_mem: Stores output stack in memory i.e. temporary storage
- in_disk: Stores output stack on hard-disk drive i.e. permanent storage
Stack writing instruction:
- stack_replace: to replace the complete storage-stack or to create a new storage-stack
- stack_append: to append fetched values to a pre-existing storage-stack
- key_replace: to replace all values of pre-existing key/column in a storage-stack with new values
Using the in_mem option, you can write the result set for a query to a temporary filenode in the memory of the DNIF-container. This storage-stack is only available till the container or the host-machine restarts.
_fetch * from event where $Duration=12h AND $GivenName=App group count_unique $Action limit 100 >>_store in_mem action_list stack_replace
The above query will fetch distinct $Action from the machine whose $GivenName is ‘App’, and its result set will be pipelined to the directive, _store, to store the result in_mem with the store-name as ‘action_list’.
To retrieve or view the data stored in a storage-stack, please refer the _retrieve - Read from Filenode document.
Using the in_disk option, you can write a result set on a permanent filenode on the DNIF-container’s Hard Disk Drive [HDD]. This storage-stack will remain available unconditionally to the container or host-machine until restart
_fetch * from event where $Duration=12h AND $GivenName=App group count_unique $Action limit 100 >>_store in_disk action_list stack_replace
The above query will fetch distinct $Action from the machine whose $GivenName is ‘App’, and its result set will be pipelined to the directive, _store, to store the result in_disk, the DNIF HDD, with the store-name as ‘action_list’.
To retrieve or view the data stored in a storage-stack please refer the _retrieve - Read from Filenode document.
Stack Writing Instruction:
This writing instruction is commonly used while creating a new store, either in memory or on disk. Running the stack_replace instruction on pre-existing stack will truncate and overwrite it with the latest output.
To further understand this concept, let’s create a fresh in_disk stack:
_fetch * from event where $Duration=1h AND $LogType=WEBSERVER group count_unique $SrcIP limit 10 >> _store in_disk src_list_web stack_replace
To check entries that have been stored, let’s retrieve the stored stack, src_list_web, from the previous query:
As Shown above, we’ve successfully created a storage-stack, src_list_web, that contains a list of distinct source IPs, $SrcIP, that had connected to our web server in the last 1 hour, with their associated event counts in the count_unique column.
The key_replace writing instruction is useful when there is a need to update the data stored with the latest search values for certain pre-existing keys. This instruction will automatically append a new key to the store.
Using the above example, let’s execute the following query on the store, src_list_web:
_fetch * from event where $Duration=5h AND $LogType=WEBSERVER group count_unique $SrcIP limit 5 >> _store in_disk src_list_web key_replace $SrcIP
If you’ve noticed, we’ve added $SrcIP in the end of the _store query. Here, the field $SrcIP is a key for which we want the updated results to be fetched from the _fetch query.
Let’s now retrieve the updated store, src_list_web, from the previous query.
As you can see from the highlights above, as the result for _retrive, the value of count_unique for the three IPs were updated, as they were already in the store. Besides, two more entries were added at the bottom of the list.
So, in a nutshell, key_replace will update the relevant values for a given key if it is already present in the stack. In case of a new entry, key_replace will append the key-value pair.
Unlike key_replace, this instruction can only append the fetched result at the bottom of the stack
Continuing with the same example of the src_list_web store getting updated by the key_replace query, here’s how stack_append would work:
_fetch * from event where $Duration=3h AND $LogType=WEBSERVER group count_unique $SrcIP limit 2 >> _store in_disk src_list_web stack_append
In the query above, we are retrieving the list of active IP addresses from our web server for the last 3 hours, and are adding them to the existing store, src_list_web, using stack_append.
Now, let’s _retrieve the store src_list_web:
In the above screenshot, highlighted in red, we see the newly added results at the bottom of the stack, whereas similar IP addresses were already available in the store.