April 02, 2017 / by jburks / In guides /

How To Integrate OSQuery And DNIF

The following sections will take you through the integration of OSQuery with DNIF.

OSQuery

It’s an OS instrumental framework that can work with Linux, Windows, OS X(macOS) and FreeBSD platforms. It will perform a low-level monitoring and analytics on both performant and intuitive.

It’s a cross-platform application with support for recent versions of macOS, Windows 10, CentOS, and Ubuntu.

It’s officially described as an “SQL-powered operating system instrumentation, monitoring, and analytics” framework, and originated from Facebook.

This tool is widely adopted for:

  • State-based change detection.
  • File integrity monitoring
  • Process auditing
  • Socket auditing

Getting started with OSQuery

Installing osquery gives you access to the following components:

osqueryi: The interactive osquery shell, for performing ad-hoc queries.

osqueryd: A daemon for scheduling and running queries in the background service mode.

osqueryctl: A helper script for testing a deployment or configuration of osquery.

It can also be used instead of the operating system’s service manager to start/stop/restart osqueryd.

osqueryi and osqueryd completely independent tools. They won’t communicate with each other, and you can use one without impacting the other. Whereas, most of command-line-flags and options needed to run on each are same, and you can run osqueryi using osqueryd’s configuration file in order to customize the launch environment without using lots of command-line-switches and flags.

To start with osquery, you need:

  • osquery installed on a host machine. logging infrastructure [dnif:Adapter component].

  • Log forwarder installed on a host machine.

Installing OSQuery on Windows Server

Osquery recommends to install and deploy Windows support using chocolatey.
Install command:

C:\> choco.exe install osquery

For more details on chocolatey project, please visit Chocolatey Project Page.

How to install Chocolatey
  • Requirments:
    • Administrator Access
    • Windows 7+ / Windows Server 2003+
    • PowerShell v2+
    • .NET Framework 4+
      [Note: the default installation attempts to install .NET 4.0 if you don’t have it previously installed]

Note: You need to run below mentioned commands as an ‘Administrator’.

Installing Chocolatey Reference Document:
https://chocolatey.org/install

Using ‘cmd.exe’ console:

C:> @powershell -NoProfile -ExecutionPolicy Bypass -Command "iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET "PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin"

Output will look something like:

Install_Choco

Upgrade Chocolatey to latest version:

C:> choco upgrade chocolatey

Output:

Upgrade_Choco

Installing osquery using chocolatey:
https://chocolatey.org/packages/osquery

Installing osquery using ‘choco.exe’ :

C:> choco install osquery

Output will look something like:

Install_Osquery

By default choco.exe chocolatey will install the binaries, example packs and configurations bundle to c:\ProgramData\osquery. osqueryd has not been installed as a service yet. To install this, you can either pass Chocolatey the --params='/InstallService' flag during install command i.e.:

C:\> choco.exe install osquery --params='/InstallService'

OR,
you can make use of osquery’s --install flag with below given command to install osqueryd as a Windows system service.

C:\ProgramData\osquery\osqueryd\osqueryd.exe --install
OSQry_as_Service_cmd

You can also create osquery-service-daemon using Powershell manage-osqueryd.ps1 script in C:\ProgramData\osquery folder. This same script can also be used to start and stop the osqueryd service.

PS_manage_service_cmd

Creating osquery configuration file

Creating configuration file makes it easier to run osqueryi and osqueryd. Insted of passing lots of command line options, both osqueryi and osqueryd can read those options from a configuration file from file-path C:\ProgramData\osquery\osquery.conf.

There are three sections to the configuration file:

  • A list of daemon service options
  • A list of scheduled queries to run and when they should run.
  • A list of packs to include more specific scheduled queries.

Create and open the configuration file at: C:\ProgramData\osquery\osquery.conf

Note: make sure that created file had proper extention as osquery.conf and not osquery.conf.txt.

The configuration file uses the JSON format. Copy the following content into the file:

{
  "options": {
    "config_plugin": "filesystem",
    "logger_plugin": "filesystem",
    "disable_logging": "false",
    "log_result_events": "true",
    "schedule_splay_percent": "10",
    "events_expiry": "3600",
    "events_max": "2000",
    "verbose": "false",
    "worker_threads": "2",
    "logger_mode": "420",
    "enable_monitor": "true",
    "disable_events": "false",
    "events_optimize": "true",
    "host_identifier": "hostname",
    "schedule_default_interval": "3600",
    "disable_distributed": "true"
  },
  "schedule": {
    "winevnt": {
      "query": "select * from windows_events;",
      "interval": 10
    }
  },
  "decorators": {
    "load": [
      "SELECT uuid AS host_uuid FROM system_info;",
      "SELECT user AS username FROM logged_in_users ORDER BY time DESC LIMIT 1;"
    ]
  },
  "packs": {
     "osquery-monitoring": "/usr/share/osquery/packs/osquery-dnif-linux.conf"
  }
}

Save and close the file, then validate it using the following PowerShell command:
PS C:\ProgramData\osquery\osqueryd> .\osqueryd.exe -config_check If there’s an error, the output will indicate the location of the error so you can fix it.


Running osqueryd

Once the configuration file is in place, we can start osqueryd using any of below given methods:

  1. Using cmd.exe
    Start-Service osqueryd

  2. Using PowerShell
    sc.exe start osqueryd OR PS C:\ProgramData\osquery> .\manage-osqueryd.ps1 -start

  3. Using Run–>service.msc
    Locate service ‘osquery service daemon’ and start it.

service_msc_SShot

Installing osquery on Linux server

Building osquery from source

Osquery can also be compiled and installed from the source code. For this, you have to download the source from git repository.

Dependencies:
  • sudo
  • make [This has to be GNU make]
  • python
  • ruby
  • git
  • bash

Please note that default source package comes with command make deps to install or build above dependencies. Somehow, it also installs various packages that are not required to “use” osquery, else only required to build osquery binaries and packages on a host machine.

$ git clone http://github.com/facebook/osquery.git  
$ cd osquery  
$ make deps  
$ make -j 8  
$ ./build/<platform>/osquery/osqueryi

For reference page on osquery site, click here.


Installation on Ubuntu/CentOS/RHEL machine

Apart from source compilation as mentioned above. There are two additional methods to install osquery on Ubuntu/CentOS/RHEL Linux machine:

  1. Using pre-built binary ‘.deb’ or ‘.rpm’ package. or;
  2. from apt and yum repository
1. From ‘.deb’ or ‘.rpm’ package.

You can download a pre-built osquery DEB or RPM package that contains osquery binaries, init.d script and example configurations.
Using dpkg -i {.deb package} command in Ubuntu and rpm -i {.rpm package} command in CentOS/RHEL, you can install it in host machine.

2. From ‘apt’ and ‘yum’ repository

There is no installable package for osquery in the official Ubuntu repository. You will have to add project repository to system.

Ubuntu 16.04 LTS Xenial

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B  
$ sudo add-apt-repository "deb [arch=amd64] https://osquery-packages.s3.amazonaws.com/xenial xenial main"  
$ sudo apt-get update  
$ sudo apt-get install osquery

Ubuntu 14.04 & 12.04 LTS

$ sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 1484120AC4E9F8A1A577AEEE97A80C63C9D8B80B  
$ sudo add-apt-repository "deb [arch=amd64] https://osquery-packages.s3.amazonaws.com/trusty trusty main"  
$ sudo apt-get update  
$ sudo apt-get install osquery

CentOS/RHEL 7.0

$ sudo rpm -ivh https://osquery-packages.s3.amazonaws.com/centos7/noarch/osquery-s3-centos7-repo-1-0.0.noarch.rpm  
$ sudo yum install osquery

CentOS/RHEL 6.6

$ sudo rpm -ivh https://osquery-packages.s3.amazonaws.com/centos6/noarch/osquery-s3-centos6-repo-1-0.0.noarch.rpm  
$ sudo yum install osquery

For download page on osquery site, click here.


Creating osquery configuration file

Creating configuration file makes it easier to run osqueryi and osqueryd. Instead of passing lots of command line options, both osqueryi and osqueryd can read those options from a configuration file located in /etc/osquery/osquery.conf.

There are three sections to the configuration file:

  • A list of daemon service options
  • A list of scheduled queries to run and when they should run.
  • A list of packs to include more specific scheduled queries.

Create and open the configuration file using following command:
$ sudo nano /etc/osquery/osquery.conf

The configuration file uses the JSON format. Copy the following content into the file:

{
  "options": {
    "config_plugin": "filesystem",
    "logger_plugin": "filesystem",
    "logger_path": "/var/log/osquery",
    "disable_logging": "false",
    "log_result_events": "true",
    "schedule_splay_percent": "10",
    "pidfile": "/var/osquery/osquery.pidfile",
    "events_expiry": "3600",
    "database_path": "/var/osquery/osquery.db",
    "verbose": "false",
    "worker_threads": "2",
    "enable_monitor": "true",
    "disable_events": "false",
    "disable_audit": "false",
    "audit_allow_config": "true",
    "host_identifier": "hostname",
    "enable_syslog": "false",
    "audit_allow_sockets": "true",
    "schedule_default_interval": "3600"
  },
  "schedule": {
    "usb_ports": {
      "query": "select *,'Removable USB device detected.' as msg from usb_devices where removable=1 ;",
      "platform": "linux",
      "interval": 10
    },
    "local_listning_ports": {
      "query": "select *, 'Change in local listning port detected' as msg from listening_ports inner join processes on  listening_ports.pid = processes.pid where address='0.0.0.0' ;",
      "platform": "linux",
      "interval": 60
    },
    "local_processes": {
      "query": "select *, 'Local process start-stop detected' as msg from processes where on_disk=1;",
      "platform": "linux",
      "interval": 60
    }
  },
  "decorators": {
    "load": [
      "SELECT 'dnif_osquery' as log_type;",
      "SELECT uuid AS host_uuid, hardware_model FROM system_info;",
      "SELECT user AS username FROM logged_in_users ORDER BY time DESC LIMIT 1;"
    ]
  }
}

Save and close the file, then validate it using the following command:
$ sudo osqueryctl config-check
If there’s an error, the output will indicate the location of the error so you can fix it.


Running osqueryd

osqueryd daemon allows osquery to run queries set in intervals. Those queries include the ones you configured in the above step.
Results generated by osqueryd are written to a file called osqueryd.results.log in the /var/log/osquery directory. By default this file won’t exist. It only gets created when the daemon is started and starts generating results.

You can start osqueryd using either systemctl or osqueryctl. Both accomplish the same thing, so it doesn’t matter which one you use. osqueryd will check for the existence of a configuration file when it starts, and alert you if it doesn’t find one. It will remain running without a configuration file, although it won’t do anything useful.

Once the configuration file is setup as per above instructions, you can start the daemon with the following command:
$ sudo systemctl start osqueryd
Or you can type:
$ sudo osqueryctl start