LogSentinel Collector Overview¶
The LogSentinel Collector is an open source component that gets installed on-premise to listen to a configured set of log sources. It can be installed on Linux and Windows and supports the following types of sources:
- Log files an arbitrary text file can be collected and sent, line by line, to the LogSentinel SIEM service. These typically application logs and logs by systems like application servers, accounting software, CRMs, ERPs (e.g. SAP), big data processing, or any other non-standard software
- Database tables - if you store audit trail inside relational database tables, you can configure queries that periodically fetch new entries and send them to the LogSentinel SIEM service
- Syslog - collect syslog messages sent by any appliances pointed to the collector, in any format and variation (RFC 3164, RFC 5424; CEF, LEEF)
- NetFlow/IPFIX/sFlow - act as a NetFlow/IPFIX/sFlow collector and forward the flows to the SIEM
- SSH - connect to any system over SSH and listen to text files or the activities of the logged in users
- Database logs files - if database query logs are enabled, the collector listens to newly issued queries and sends them to the LogSentinel SIEM service
- MS SQL audit trail – if MS SQL audit trail is enabled, the collector can be configured to listen to it and forward the events
- MS SQL change tracking – if MS SQL change tracking is enabled, the collector can be configured to listen to it and forward the change events
- Oracle audit trail – if Oracle audit trail is enabled, the collector can be configured to listen to it and forward the events
- Access logs – the standard web server access log files can be parsed (in Common Log Format and Extended Log Format) and sent to LogSentinel SIEM
- Linux audit log – the native linux audit log file can be tailed and forwarded to LogSentinel SIEM
- Windows event logs – the Windows event logs (including all categories – Application, Security, System) can be read continuously and sent to LogSentinel SIEM
- Directory changes - any changes in a directory (new files, removal of files, modification of files) can be tracked using this collector configuration.
- PostgreSQL - the collector can collect the audit logs generated by pgaudit
- MySQL - the collector can collect the audit logs generated by the audit log plugin
- Teradata - if auditing is enabled on Teradata, the collector can query the logs and forward them
- Hadoop - the collector can parse and forward Hadoop security logs
- AxonDB logs – AxonDB is a special type of non-relational database. We support its custom log format.
- VSphere logs – the VCenter events and the system events logs (if HostHealthStatusSystem is enabled) can be tracked using this collector configuration.
- SNMP traps - the collector can act as an SNMP manager and receive SNMP traps
Any combination of the above can be configured. Note that for most target types the collector can be installed on a different machine than the actual log source, thus supporting full agentless collection. For example, in case of database tables or database audit trail, the collector can be installed on another machine that connects to the database server via a database connection string and credentials.
All communication between the collector and the LogSentinel SIEM is encrypted .
Configuration and installation is done through scripts provided by us and you can follow the steps here.
In order to find existing logs and tables to monitor and send to LogSentinel SIEM, you can use our simple open source scan-logs script tool.
Collecting remote files¶
Text files residing on servers other than the one where the LogSentinel Collector is installed is typically a challenge for log collection. Please refer to our Collecting Remote Files section for more details.
In case of lack of connectivity to the SIEM, the connector can keep a queue of the data or keep trying to read the same data which it failed sending, for an extensive period of time. After connection is restored, the unsent logs are sent in batches to the SIEM.
Queues are kept in memory (for syslog and netflow sources) until a certain limit is reached and then are stored in disk. While collectors usually don't require much disk space, such scenarios should be taken into account when provisioning collector disk space.
Custom connectors can be easily added to LogSentinel Collector. This can be done in two ways:
- Python scripts to extend a basic Python executor, useful to parse non-standard formats for text files, syslog messages, etc.
- Java extensions - our collector is open-source and can be extended (by us or by a partner) to accomodate any use-case.