LogSentinel Collector Configuration¶
Configuration via UI¶
The LogSentinel Collector exposes a web-based UI on port 8070 that allows you to configure multiple sources quickly. The UI is sycnrhonized with the YAML configuration so you don't have to choose one over the other
Configuration file¶
Below is a full reference of the configuration options for the LogSentinel Collector. It can read
# Data source ID (ApplicationId), OrganizationId and secret obtained from the API credentials page in the dashboard.
# The dataSourceId can be overridden per targetType
dataSourceId: ba2f0680-5424-11e8-b88d-6f2c1b6625e8
organizationId: ba2cbc90-5424-11e8-b88d-6f2c1b6625e8
secret: d8b63c3d82a6deb56b005a3b8617bf376b6aa6c181021abd0d37e5c5ac9911a1
# The type of event being sent by default. Options are: BUSINESS_LOGIC_ENTRY, DATABASE_QUERY, SYSTEM_EVENT.
# It is normally a good idea to not change this field
entryType: BUSINESS_LOGIC_ENTRY
# Supported collector types (all of them except syslog, netflow, honeypot and ossec support a list of entries):
# file - watches one or more files and sends each new line as a separate event
# ssh - watches remote files over SSH
# sshUser - watches remote server user activity using the "w" command
# database - watches one or more tables using custom queries and sends events based on a comparison column (usually timestmap or sequential ID)
# databaseLog - watches a database query log and sends each query as a seprate event
# directory - watches a given directory for changes (files created, deleted or modified)
# accessLogFile - watches an access log file and parses the standard access log format
# mssqlAuditLog - watches MS SQL Server audit log (needs to be properly configured prior to starting the collector)
# mssqlChangeTracking - watches MS SQL Server change tracking details and sends each change
# mssqlEventLog - watches the MS SQL Server Windows logs
# mssqlLogin - watches for MSSQL Server login events
# linuxAuditLog - watches and parses the default Linux audit log file
# windowsEventLog - watches Windows event log and sends each entry
# exchangeAdminLog - watches the admin log of Microsoft Exchange
# leakedCredentials - watches mail server for email addresses to send for leaked credentials monitoring
# axonDb - used to interact with AxonDB modifications to turn them into audit trail entries
# syslog - used to activate a syslog server that forwards syslog events to LogSentinel
# netflow - used to activate a netflow v9 collector (partially compatible with IPFIX)
# snmp - used to receive SNMP traps
# sap - monitors directory for and handles SAP Security audit logs, which happen to be separated by a special separator (all logs are on the same line)
# oracle - configures auditing and FGA on Oracle DB and watches the DBA_AUDIT_TRAIL and FGA_LOGS tables (extending the database collector)
# postgre - watches a PostgreSQL database using pgaudit
# mysql - watches a MySQL database using the audit log plugin
# honeypot - acts as a decoy server on pre-defined services, prtocols and ports collecting credentials from malicious attempts
# vsphere - watches VCenter events and system event logs
# ossec - receives messages from the OSSEC endpoint agent
# vaultAuditLog - listens for HashiCorp Vault audit log messages
# The base URL to connect to. Change only for on-premise deployments
logsentinelBaseUrl: https://api.logsentinel.com
# Keystore configurations. Use only if you need each request to be digitally signed.
keystorePath: /path/to/keystore.jks
keystorePassword: password
keystoreAlias: alias
# Configure whether the MAC address of the machine is send as a parameter attached to each event
includeMacAddress: false
# Configure whether the local IP address of the machine is send as a parameter attached to each event
includeLocalIp: false
# Configure whether collectors that rely on timestamp to send events should start from events that
# happen after the collector is installed, or historical events should be consumed and sent as well
# (not applicable if historical data is not available)
timestampInitialUseCurrent: true
# Allows trusting self-signed certificates provided by the LogSentinel service.
# Use only for on-premise installations
trustSelfSignedCertificates: false
# Allows authentication of users
authentication:
enabled: true
jwtSecret: testtesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttesttest
jwtSecretVaultKey:
usernames:
- TestUsername
file:
- dataSourceId: ... # override the default dataSourceId to send events to a custom one
watchIntervalMillis: 30000 # period for checking the files for updates
watchFilePaths: # list of files to watch
- /var/logs/system.log
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
separator: # if the file has columns, specify the column separator (;,\t)
csv: #true or false, whether the CSV syntax should be observed, e.g. escaping quotes;
actionIdx: #0-based index of the action column
actorIdx: #0-based index of the actor column
entityTypeIdx: #0-based index of the entityType column
entityIdIdx: #0-based index of the entityId column
compressionType: #GLIB or ZLIP - used for reading compressed audit log files
ssh:
- dataSourceId: c04bbd80-219e-11ea-bc18-c5a6448d7eee
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
host: ... # the host (ip or hostname) to connect to
port: 22 # port to connect to, specify only if non-default is used
username: ... # ssh username
password: ... # ssh password
privateKey: ... # path the a private key (if needed)
privateKeyPassword: ... # password for the private key
fileConfig: # file config nested obect, following the definition for "file" above:
accessLogFileConfig: ... # optional access log file parser config, following the "accessLogFile" definition
xmlFileConfig: ... # optional XML log file parser config, following the "xml" definition
sshUser:
- dataSourceId: c04bbd80-219e-11ea-bc18-c5a6448d7eee
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
host: ... # the host (ip or hostname) to connect to
port: 22 # port to connect to, specify only if non-default is used
username: ... # ssh username
password: ... # ssh password
privateKey: ... # path the a private key (if needed)
privateKeyPassword: ... # password for the private key
databaseLog:
- dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
watchIntervalMillis: 30000 # period for checking the files for updates
watchFilePaths: # list of files to watch
- /var/logs/system.log
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
linuxAuditLog:
- dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
watchIntervalMillis: 30000 # period for checking the files for updates
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
accessLogFile:
- dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
# the access log format in Common Log Format - https://en.wikipedia.org/wiki/Common_Log_Format
accessLogFormat: format
accessLogIgnoredPaths: # option to ignore requests to a list of URIs
- path1
- path2
watchIntervalMillis: 30000 # period for checking the files for updates
watchFilePaths: # list of files to watch
- /var/logs/system.log
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
directory:
- watchDirPath: /var/logs # directory to watch for changes
sendLogsRate: 30000
dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
useDocumentApi: false # whether to use the document API (more conventient for tracking documents)
sendHash: false # whether to send just the hash of the file rather than the whole body
maxFileSize: 0 # the max allowed file size to send to the server; otherwise a hash is sent; 0 means no limit (not that there's a server limit)
skipActorId: false # whether no attempt is made to extract the actorId for each even (auditing functionality must be turned on)
sendInBatches: true # whether to send the data in batches or in real time as events come
database:
- jdbcConnectionString: jdbc:mysql://192.168.1.101/db # database connection string
jdbcUsername: root # database username
jdbcPassword: pass # database password
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
watchSqlQueries: # list of queries to be executed against the database
- sql: select * from logs # SQL query
# which column is used for comparing entries. Only entries with value
# of this column above the value of the last sent event will be processed
criteriaColumn: timestamp
actorDisplayNameColumn: actorDisplayName # column to get the actorDisplayName
actorIdColumns: actorId # comma separated columns that comprise the actorId
actionColumn: action # column to be used for the action
entityIdColumn: entityId # column to be used for entityId
entityTypeColumn: entityType # column to be used for entityType
entityTypeValue: entityType # a hardcoded value for entityType (alternative to specifying a column)
actionValue: action # a hardcoded value for the action (alternative to specifying a column)
- sql: select * from events
criteriaColumn: timestamp2
actorDisplayNameColumn: actorDisplayName2
actorIdColumns: actorId2 #comma separated
actionColumn: action2
entityIdColumn: entityId2
entityTypeColumn: entityType2
entityTypeValue: entityType2
actionValue: action2
oracle:
- jdbcConnectionString: jdbc:oracle:thin:@192.168.1.110:1521:orcl # database connection string
jdbcUsername: SYS AS SYSDBA # database username
jdbcPassword: pass # database password
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
fgaPolicies: # Fine Grained Auditing policies
- objectSchema: TESTUSER #schema
objectName: PERSONS #table name
policyName: testPolicy #policy name, necessary
auditCondition: Age != 0,
auditColumn: City,
enabled: true
statementTypes: SELECT, INSERT, UPDATE, DELETE
auditPolicies: # Standard audit policies
# available audit options see with SELECT * FROM SYS.STMT_AUDIT_OPTION_MAP
- userName: testUser # user to which the policy applies, leave empty to apply for all users
auditOption: UPDATE TABLE # Action that will be audited
- auditOption: SELECT TABLE
vSphere:
- name: # a human-readable name for this config, useful when multiple configurations per type
username: # vCenter username
password: # vCenter password
serverName: # vCenter server name (IP or FQDN)
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
windowsEventLog:
- sourceTypes: # list of windows event log types (Application, Security, System)
- Application
- Security
sources: # an optional whitelist of event log sources to be processed.
- Source1
- Source2
excludedSources: # a blacklist of event sources not to be processed. Alternative to specifying "sources"
- ExecludedSource1
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
mode: NATIVE # Defines whether to use the NATIVE Win32 API (default) or PowerShell commands (specified by POWERSHELL value)
remote: # connect to a remote event log. Leave user, domain and password blank to use current user
user:
domain:
password:
server: # ip address or domain name
authMethod: # auth constant, starting from 0 https://docs.microsoft.com/en-us/windows/win32/api/winevt/ne-winevt-evt_rpc_login_flags
dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
exchangeAdminLog:
- exchangeUrl: # url of the exchange server
username: # username to connect with
password: # password to connect with (if username and password are not specified, the currenet account is used)
dateFormat: # override the default date format if needed in order to parse the dates
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
mssqlAuditLog:
- jdbcConnectionString: jdbc:mysql://192.168.1.101/ # database connection string
jdbcUsername: sa # database username
jdbcPassword: pass # database password
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
dataSourceId: ... # override the default dataSourceId to send events for
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
# path where the MS SQL Server audit log file is stored.
# See https://github.com/LogSentinel/logsentinel-agent/blob/master/MS_SQL_README.md)
mssqlLogsPath: c:\auditlog\
mssqlChangeTracking:
- jdbcConnectionString: jdbc:mysql://192.168.1.101/ # database connection string
jdbcUsername: sa # database username
jdbcPassword: pass # database password
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
dataSourceId: ... # override the default dataSourceId to send events to a
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
databases: # list of databases for which changes have to be tracked
- db1
- db2
# tables for which changes should be ignored (use the full table name, including database and schema)
ignoredTables:
- db1.dbo.table1
- db2.dbo.table3
# tables for which changes should be monitored. If not specified, all tables are monitored.
includedTables:
- db1.dbo.table1
- db2.dbo.table3
mssqlEventLog:
- ... # same properties as windowsEventLog
mssqlLogin:
- jdbcConnectionString: jdbc:mssql://192.168.1.101/db # database connection string
jdbcUsername: root # database username
jdbcPassword: pass # database password
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple
includedUsers: ... # a list of users to include (the rest are ignored)
compactDuplicateLogins: false # whether duplicate login events should be ignored
axonDB:
- trackingToken: 0 # AxonDB tracking token
action: LOG_AXON # Hardcoded action value
batchEnabled: false # Use batch queries
batchInterval: 10000 # Batch interval in case batch queries are enabled
dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
leakedCredentials:
- username: # username used to connect to Exchange server
password: # password used to connect to Exchange server
groupName: # group name from which emails will be obtain
ldapDN: #
ldapProviderUrl: # url of the ldap server
ldapPrincipal: # DN if ldap server doesn't allow reading for anonymous users
ldapPassword: # password ldap server doesn't allow reading for anonymous users
ldapEnabled: false # 'true' when using LDAP, 'false' when using Exchange server
sendLogsRate: 259200000 # how often is data sent to the LogSentinel service (3 days)
email:
#imap url - replace <username> and <password> with real ones. Replace imap.gmail.com for other providers
# Make sure Imap server accepts requests from outside (gmail does not by default)
imapInboxUrl: imap://<username>:<password>@imap.gmail.com/INBOX
actionRegex: .* # regex extracting action from subject or body
actionRegexSubject: true # search action in subject or body
entityIdRegex: .* # regex extracting entityId from subject or body
entityIdRegexSubject: false # search entityId in subject or body
entityTypeRegex: .* # regex extracting entityType from subject or body
entityTypeRegexSubject: false # search entityType in subject or body
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
syslog:
hostToDataSourceId: # a map of source IP(s) to data source id(s). Use "default" to match "any IP"
hostTagConcatenation: # whether the TAG field should be appended to the details for RFC3164 syslog messages, specify a boolean value per host. False by default.
netFlow: # configuration for the NetFlow (v9) connector
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
port: 2055 # the port at which the NetFlow v9 collector listens (default port for NetFlow is 2055)
hostToDataSourceId: # a map of source IP(s) to data source id(s). Use "default" to match "any IP"
"127.0.0.1": <some-data-source-id>
"default": <other-data-source-id>
snmp:
hostToDataSourceId: # a map of source IP(s) to data source id(s). Use "default" to match "any IP"
port: 162 # change to override the default port
sapSecurityAuditLog:
- watchIntervalMillis: 30000 # period for checking the files for updates
watchFilePaths: # list of files to watch
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
separator: # specify the special log separator
compressionType: #GLIB or ZLIP - used for reading compressed audit log files
sapReadAccessLog:
- jdbcConnectionString: jdbc:mssql://192.168.1.101/db # database connection string
jdbcUsername: root # database username
jdbcPassword: pass # database password
sendLogsRate: 30000 # how often is data sent to the LogSentinel service
dataSourceId: ... # override the default dataSourceId to send events to a custom one
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
cdc:
# Full documentation for supported databases can be found here:
# https://debezium.io/docs/connectors/mysql/
# https://debezium.io/docs/connectors/postgresql/
# https://debezium.io/docs/connectors/oracle/
# https://debezium.io/docs/connectors/sqlserver/
# regexes to extract actor, action, entityId and entityType from stringified data provided by debezium
- actorRegex: .*
actionRegex: .*
entityIdRegex: .*
entityTypeRegex: .*
name: ... # a human-readable name for this config, useful when multiple configurations per type are used
# supported databases are MYSQL, ORACLE, POSTGRES, MSSQL
database: MYSQL
offsetFilename: offset.txt #path to file that stores current processed state
databaseHost: localhost
databasePort: 3306
databaseDbname: test # database name
databaseUser: user
databasePassword: password
databaseServerName: serverName # logical name, used to distingush different debezium instances (if any)
databaseHistoryFilename: history.txt # file where the connector will write and recover DDL statements
tableWhiteList: table1 # tables that will be monitored (only for some databases)
# Additional properties supported by debezium can be placed here. Not supported keys are ignored
# Can override existing hardcoded values of properties offset.flush.interval.ms and server.id
additionalProperties:
key1: value1
postgre:
- name: ... # a human-readable name for this config, useful when multiple configurations per type are used
jdbcConnectionString: jdbc:postgresql://xxx.xxx.xxx:5432/postgres # the JDBC connection string
jdbcUsername: ... # postgre username
jdbcPassword: ... # postgre password
sendLogsRate: 30000 # how often is data sent to the LogSentinel service in millis
pgauditEnabled: ... # whether pgaudit is required or the native log file is used
schemas: # schemas to monitor for audit logs
- public
excludedTables: # an optional list of tables to exclude from processing
- ...
honeypot:
dataSourceId: ... # override the default dataSourceId to send events to a custom one
testMode: false # testMode disables fetching scanner IPs(mainly for performance) and doesn't add firewall blocking rules
ignoredCIDR: ... # list of CIDR which wont be blocked if attempted a connection
protocols:
- SSH
- HTTPS
- HTTP
- RDP
- SMB
- FTP
ossec:
hostToDataSourceId:
"ip": ...
sendLogsRate: 30000
queueSize: 5
udpPort: 1514
agentKeys:
agentId: key
vaultAuditLog:
- port: 9090 # port to listen to
name: testing # human-readable name
dataSourceId: ... # the id of the data source to send data for