LogSentinel Collector Configuration

Configuration via UI

The LogSentinel Collector exposes a web-based UI on port 8070 that allows you to configure multiple sources quickly. The UI is sycnrhonized with the YAML configuration so you don't have to choose one over the other

Configuration file

Below is a full reference of the configuration options for the LogSentinel Collector. It can read

# Data source ID (ApplicationId), OrganizationId and secret obtained from the API credentials page in the dashboard. 
# The dataSourceId can be overridden per targetType
dataSourceId: ba2f0680-5424-11e8-b88d-6f2c1b6625e8
organizationId: ba2cbc90-5424-11e8-b88d-6f2c1b6625e8
secret: d8b63c3d82a6deb56b005a3b8617bf376b6aa6c181021abd0d37e5c5ac9911a1

# The type of event being sent by default. Options are: BUSINESS_LOGIC_ENTRY, DATABASE_QUERY, SYSTEM_EVENT. 
# It is normally a good idea to not change this field
entryType: BUSINESS_LOGIC_ENTRY


# Supported collector types (all of them except syslog, netflow, honeypot and ossec support a list of entries):
  # file - watches one or more files and sends each new line as a separate event
  # ssh - watches remote files over SSH
  # sshUser - watches remote server user activity using the "w" command
  # database - watches one or more tables using custom queries and sends events based on a comparison column (usually timestmap or sequential ID)
  # databaseLog - watches a database query log and sends each query as a seprate event 
  # directory - watches a given directory for changes (files created, deleted or modified) 
  # accessLogFile - watches an access log file and parses the standard access log format
  # mssqlAuditLog - watches MS SQL Server audit log (needs to be properly configured prior to starting the collector)
  # mssqlChangeTracking - watches MS SQL Server change tracking details and sends each change
  # mssqlEventLog - watches the MS SQL Server Windows logs
  # mssqlLogin - watches for MSSQL Server login events
  # linuxAuditLog - watches and parses the default Linux audit log file 
  # windowsEventLog - watches Windows event log and sends each entry
  # exchangeAdminLog - watches the admin log of Microsoft Exchange
  # leakedCredentials - watches mail server for email addresses to send for leaked credentials monitoring
  # axonDb - used to interact with AxonDB modifications to turn them into audit trail entries 
  # syslog - used to activate a syslog server that forwards syslog events to LogSentinel
  # netflow - used to activate a netflow v9 collector (partially compatible with IPFIX)
  # snmp - used to receive SNMP traps
  # sap - monitors directory for and handles SAP Security audit logs, which happen to be separated by a special separator (all logs are on the same line)
  # oracle - configures auditing and FGA on Oracle DB and watches the DBA_AUDIT_TRAIL and FGA_LOGS tables (extending the database collector)
  # postgre - watches a PostgreSQL database using pgaudit
  # mysql - watches a MySQL database using the audit log plugin
  # honeypot - acts as a decoy server on pre-defined services, prtocols and ports collecting credentials from malicious attempts
  # vsphere - watches VCenter events and system event logs 
  # ossec - receives messages from the OSSEC endpoint agent
  # vaultAuditLog - listens for HashiCorp Vault audit log messages 

# The base URL to connect to. Change only for on-premise deployments
logsentinelBaseUrl: https://api.logsentinel.com

# Keystore configurations. Use only if you need each request to be digitally signed.  
keystorePath: /path/to/keystore.jks
keystorePassword: password
keystoreAlias: alias

# Configure whether the MAC address of the machine is send as a parameter attached to each event
includeMacAddress: false

# Configure whether the local IP address of the machine is send as a parameter attached to each event
includeLocalIp: false

# Configure whether collectors that rely on timestamp to send events should start from events that 
# happen after the collector is installed, or historical events should be consumed and sent as well 
# (not applicable if historical data is not available) 
timestampInitialUseCurrent: true

# Allows trusting self-signed certificates provided by the LogSentinel service. 
# Use only for on-premise installations
trustSelfSignedCertificates: false

# Allows authentication of users
authentication:
  enabled: true
  jwtSecret: testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
  jwtSecretVaultKey:
  usernames:
      - TestUsername


file:
  - dataSourceId: ... # override the default dataSourceId to send events to a custom one
    watchIntervalMillis: 30000 # period for checking the files for updates 
    watchFilePaths: # list of files to watch
      - /var/logs/system.log
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    separator: # if the file has columns, specify the column separator (;,\t)  
    csv: #true or false, whether the CSV syntax should be observed, e.g. escaping quotes;
    actionIdx: #0-based index of the action column
    actorIdx: #0-based index of the actor column
    entityTypeIdx: #0-based index of the entityType column
    entityIdIdx: #0-based index of the entityId column
    compressionType: #GLIB or ZLIP - used for reading compressed audit log files

ssh:
  - dataSourceId: c04bbd80-219e-11ea-bc18-c5a6448d7eee
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    host: ... # the host (ip or hostname) to connect to
    port: 22 # port to connect to, specify only if non-default is used
    username: ... # ssh username
    password: ... # ssh password
    privateKey: ... # path the a private key (if needed)
    privateKeyPassword: ... # password for the private key
    fileConfig: # file config nested obect, following the definition for "file" above:
    accessLogFileConfig: ... # optional access log file parser config, following the "accessLogFile" definition 
    xmlFileConfig: ... # optional XML log file parser config, following the "xml" definition 

sshUser:
  - dataSourceId: c04bbd80-219e-11ea-bc18-c5a6448d7eee
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    host: ... # the host (ip or hostname) to connect to
    port: 22 # port to connect to, specify only if non-default is used
    username: ... # ssh username
    password: ... # ssh password
    privateKey: ... # path the a private key (if needed)
    privateKeyPassword: ... # password for the private key

databaseLog:
  - dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    watchIntervalMillis: 30000 # period for checking the files for updates 
    watchFilePaths: # list of files to watch
      - /var/logs/system.log
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service

linuxAuditLog:
  - dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    watchIntervalMillis: 30000 # period for checking the files for updates 
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service

accessLogFile:
  - dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    # the access log format in Common Log Format - https://en.wikipedia.org/wiki/Common_Log_Format
    accessLogFormat: format 
    accessLogIgnoredPaths: # option to ignore requests to a list of URIs 
      - path1
      - path2
    watchIntervalMillis: 30000 # period for checking the files for updates 
    watchFilePaths: # list of files to watch
      - /var/logs/system.log
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service


directory:
  - watchDirPath: /var/logs # directory to watch for changes
    sendLogsRate: 30000
    dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    useDocumentApi: false # whether to use the document API (more conventient for tracking documents) 
    sendHash: false # whether to send just the hash of the file rather than the whole body
    maxFileSize: 0 # the max allowed file size to send to the server; otherwise a hash is sent; 0 means no limit (not that there's a server limit)
    skipActorId: false # whether no attempt is made to extract the actorId for each even (auditing functionality must be turned on)
    sendInBatches: true # whether to send the data in batches or in real time as events come

database:
  - jdbcConnectionString: jdbc:mysql://192.168.1.101/db # database connection string
    jdbcUsername: root # database username
    jdbcPassword: pass # database password
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    watchSqlQueries: # list of queries to be executed against the database
      - sql: select * from logs # SQL query
        # which column is used for comparing entries. Only entries with value 
        # of this column above the value of the last sent event will be processed
        criteriaColumn: timestamp
        actorDisplayNameColumn: actorDisplayName # column to get the actorDisplayName
        actorIdColumns: actorId # comma separated columns that comprise the actorId
        actionColumn: action # column to be used for the action
        entityIdColumn: entityId # column to be used for entityId
        entityTypeColumn: entityType # column to be used for entityType
        entityTypeValue: entityType # a hardcoded value for entityType (alternative to specifying a column)
        actionValue: action # a hardcoded value for the action (alternative to specifying a column)
      - sql: select * from events
        criteriaColumn: timestamp2
        actorDisplayNameColumn: actorDisplayName2
        actorIdColumns: actorId2 #comma separated
        actionColumn: action2
        entityIdColumn: entityId2
        entityTypeColumn: entityType2
        entityTypeValue: entityType2
        actionValue: action2

oracle:
  - jdbcConnectionString: jdbc:oracle:thin:@192.168.1.110:1521:orcl # database connection string
    jdbcUsername: SYS AS SYSDBA # database username
    jdbcPassword: pass # database password
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    fgaPolicies: # Fine Grained Auditing policies
      - objectSchema: TESTUSER #schema
        objectName: PERSONS #table name
        policyName: testPolicy #policy name, necessary
        auditCondition: Age != 0,
        auditColumn: City,
        enabled: true
        statementTypes: SELECT, INSERT, UPDATE, DELETE
    auditPolicies: # Standard audit policies
      # available audit options see with SELECT * FROM SYS.STMT_AUDIT_OPTION_MAP
      - userName: testUser # user to which the policy applies, leave empty to apply for all users
        auditOption: UPDATE TABLE # Action that will be audited
      - auditOption: SELECT TABLE 

vSphere:
  - name: # a human-readable name for this config, useful when multiple configurations per type
    username: # vCenter username
    password: # vCenter password
    serverName: # vCenter server name (IP or FQDN)
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service

windowsEventLog:
  - sourceTypes: # list of windows event log types (Application, Security, System)
    - Application
    - Security
    sources: # an optional whitelist of event log sources to be processed. 
      - Source1
      - Source2
    excludedSources: # a blacklist of event sources not to be processed. Alternative to specifying "sources"
     - ExecludedSource1
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    mode: NATIVE # Defines whether to use the NATIVE Win32 API (default) or PowerShell commands (specified by POWERSHELL value)
    remote: # connect to a remote event log. Leave user, domain and password blank to use current user  
      user: 
      domain: 
      password:
      server: # ip address or domain name
      authMethod: # auth constant, starting from 0 https://docs.microsoft.com/en-us/windows/win32/api/winevt/ne-winevt-evt_rpc_login_flags
    dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used


exchangeAdminLog:
  - exchangeUrl: # url of the exchange server
    username: # username to connect with
    password: # password to connect with (if username and password are not specified, the currenet account is used)
    dateFormat: # override the default date format if needed in order to parse the dates
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used

mssqlAuditLog:
  - jdbcConnectionString: jdbc:mysql://192.168.1.101/ # database connection string
    jdbcUsername: sa # database username
    jdbcPassword: pass # database password
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    dataSourceId: ... # override the default dataSourceId to send events for
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    # path where the MS SQL Server audit log file is stored. 
    # See https://github.com/LogSentinel/logsentinel-agent/blob/master/MS_SQL_README.md)
    mssqlLogsPath: c:\auditlog\ 

mssqlChangeTracking:
  - jdbcConnectionString: jdbc:mysql://192.168.1.101/ # database connection string
    jdbcUsername: sa # database username
    jdbcPassword: pass # database password
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    dataSourceId: ... # override the default dataSourceId to send events to a
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    databases: # list of databases for which changes have to be tracked
      - db1
      - db2
    # tables for which changes should be ignored (use the full table name, including database and schema)
    ignoredTables: 
      - db1.dbo.table1
      - db2.dbo.table3
    # tables for which changes should be monitored. If not specified, all tables are monitored. 
    includedTables: 
      - db1.dbo.table1
      - db2.dbo.table3

mssqlEventLog:
  - ... # same properties as windowsEventLog

mssqlLogin:
  - jdbcConnectionString: jdbc:mssql://192.168.1.101/db # database connection string
    jdbcUsername: root # database username
    jdbcPassword: pass # database password
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple
    includedUsers: ... # a list of users to include (the rest are ignored)
    compactDuplicateLogins: false # whether duplicate login events should be ignored

axonDB:
  - trackingToken: 0 # AxonDB tracking token
    action: LOG_AXON # Hardcoded action value
    batchEnabled: false # Use batch queries
    batchInterval: 10000 # Batch interval in case batch queries are enabled
    dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used


leakedCredentials:
  - username: # username used to connect to Exchange server
    password: # password used to connect to Exchange server
    groupName: # group name from which emails will be obtain
    ldapDN: #
    ldapProviderUrl: # url of the ldap server
    ldapPrincipal: # DN if ldap server doesn't allow reading for anonymous users
    ldapPassword: # password ldap server doesn't allow reading for anonymous users
    ldapEnabled: false # 'true' when using LDAP, 'false' when using Exchange server
    sendLogsRate: 259200000 # how often is data sent to the LogSentinel service (3 days)

email:
  #imap url - replace <username> and <password> with real ones. Replace imap.gmail.com for other providers
  # Make sure Imap server accepts requests from outside (gmail does not by default)
  imapInboxUrl: imap://<username>:<password>@imap.gmail.com/INBOX
  actionRegex: .*  # regex extracting action from subject or body
  actionRegexSubject: true # search action in subject or body
  entityIdRegex: .* # regex extracting entityId from subject or body
  entityIdRegexSubject: false # search entityId in subject or body
  entityTypeRegex: .* # regex extracting entityType from subject or body
  entityTypeRegexSubject: false # search entityType in subject or body
  name: ... # a human-readable name for this config, useful when multiple configurations per type are used

syslog:
  hostToDataSourceId: # a map of source IP(s) to data source id(s). Use "default" to match "any IP"
  hostTagConcatenation: # whether the TAG field should be appended to the details for RFC3164 syslog messages, specify a boolean value per host. False by default.

netFlow:              # configuration for the NetFlow (v9) connector
  name: ...           # a human-readable name for this config, useful when multiple configurations per type are used
  port: 2055          # the port at which the NetFlow v9 collector listens (default port for NetFlow is 2055)
  hostToDataSourceId:   # a map of source IP(s) to data source id(s). Use "default" to match "any IP"
    "127.0.0.1": <some-data-source-id>
    "default": <other-data-source-id>

snmp:
  hostToDataSourceId: # a map of source IP(s) to data source id(s). Use "default" to match "any IP"
  port: 162 # change to override the default port

sapSecurityAuditLog:
  - watchIntervalMillis: 30000 # period for checking the files for updates 
    watchFilePaths: # list of files to watch
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    separator: # specify the special log separator
    compressionType: #GLIB or ZLIP - used for reading compressed audit log files

sapReadAccessLog:
  - jdbcConnectionString: jdbc:mssql://192.168.1.101/db # database connection string
    jdbcUsername: root # database username
    jdbcPassword: pass # database password
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service
    dataSourceId: ... # override the default dataSourceId to send events to a custom one
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used

cdc:
  # Full documentation for supported databases can be found here:
  # https://debezium.io/docs/connectors/mysql/
  # https://debezium.io/docs/connectors/postgresql/
  # https://debezium.io/docs/connectors/oracle/
  # https://debezium.io/docs/connectors/sqlserver/

  # regexes to extract actor, action, entityId and entityType from stringified data provided by debezium
  - actorRegex: .*
    actionRegex: .*
    entityIdRegex: .*
    entityTypeRegex: .*
    name: ... # a human-readable name for this config, useful when multiple configurations per type are used

    # supported databases are MYSQL, ORACLE, POSTGRES, MSSQL
    database: MYSQL
    offsetFilename: offset.txt #path to file that stores current processed state
    databaseHost: localhost
    databasePort: 3306
    databaseDbname: test # database name
    databaseUser: user
    databasePassword: password
    databaseServerName: serverName # logical name, used to distingush different debezium instances (if any)
    databaseHistoryFilename: history.txt # file where the connector will write and recover DDL statements
    tableWhiteList: table1 # tables that will be monitored (only for some databases)

    # Additional properties supported by debezium can be placed here. Not supported keys are ignored
    # Can override existing hardcoded values of properties offset.flush.interval.ms and server.id
    additionalProperties:
      key1: value1

postgre:
  - name: ... # a human-readable name for this config, useful when multiple configurations per type are used
    jdbcConnectionString: jdbc:postgresql://xxx.xxx.xxx:5432/postgres # the JDBC connection string
    jdbcUsername: ... # postgre username
    jdbcPassword: ... # postgre password
    sendLogsRate: 30000 # how often is data sent to the LogSentinel service in millis
    pgauditEnabled: ... # whether pgaudit is required or the native log file is used
    schemas: # schemas to monitor for audit logs
      - public
    excludedTables: # an optional list of tables to exclude from processing
      - ... 


honeypot:
  dataSourceId: ... # override the default dataSourceId to send events to a custom one
  testMode: false # testMode disables fetching scanner IPs(mainly for performance) and doesn't add firewall blocking rules
  ignoredCIDR: ... # list of CIDR which wont be blocked if attempted a connection
  protocols:
    - SSH
    - HTTPS
    - HTTP
    - RDP
    - SMB
    - FTP

ossec:
  hostToDataSourceId:
    "ip": ...
  sendLogsRate: 30000
  queueSize: 5 
  udpPort: 1514
  agentKeys:
    agentId: key

vaultAuditLog:
  - port: 9090 # port to listen to
    name: testing # human-readable name
    dataSourceId: ... # the id of the data source to send data for