iWebDev's blog

Each vendor has its own format for reporting event information, these event formats often lack the key information necessary to integrate the events from their devices. The ArcSight standard attempts to improve the interoperability of infrastructure devices by aligning the logging output from various technology vendors. The Common Event Format (CEF) is an open log management standard that improves the interoperability of security-related information from different security and network devices and applications. To simplify integration, the syslog message format is used as a transport mechanism. This applies a common prefix to each message, containing the date and hostname, as shown below.

Mar 16 16:35:23 host message

If an event producer is unable to write syslog messages, it is still possible to write the events to a file. To do so:

1. Omit the syslog header (show above);
2. Begin the message with the format show below

CEF:Version|Device Vendor|Device Product|Device Version|Signature ID|Name|Severity|Extension

After the mandatory CEF: prefix, the remainder of the message is formatted using a common prefix composed of fields delimited by a bar (“|”) character. The Extension part of the message is a placeholder for additional fields.

Definitions of prefix fields

Version is an integer and identifies the version of the CEF format. Event consumers use this information to determine what the following fields represent.

Device Vendor, Device Product and Device Version are strings that uniquely identify the type of sending device. No two products may use the same device-vendor and device-product pair. There is no central authority managing these pairs. Event producers have to ensure that they assign unique name pairs.

Signature ID is a unique identifier per event-type. This can be a string or an integer. Signature ID identifies the type of event reported. In the intrusion detection system (IDS) world, each signature or rule that detects certain activity has a unique signature ID assigned. This is a requirement for other types of devices as well, and helps correlation engines deal with the events.

Name is a string representing a human-readable and understandable description of the event. The event name should not contain information that is specifically mentioned in other fields. For example: “Port scan from 10.0.0.1 targeting 20.1.1.1” is not a good event name. It should be: “Port scan”. The other information is redundant and can be picked up from the other fields.

Severity is an integer and reflects the importance of the event. Only numbers from 0 to 10 are allowed, where 10 indicates the most important event.

Extension is a collection of key-value pairs. The keys are part of a predefined set. The standard allows for including additional keys as outlined under “The Extension Dictionary”. An event can contain any number of key- value pairs in any order, separated by spaces (“ “). The following example illustrates a CEF message using Syslog transport:

Mar 16 16:43:10 host CEF:0|security|threatmanager|1.0|100|trojan successfully stopped|10|src=10.0.0.192 dst=12.121.122.82 spt=1232

Character encoding

Because CEF uses the UTF-8 Unicode encoding method, please note the following:

• The entire message has to be UTF-8 encoded.
• If a pipe (|) is used in the prefix, it has to be escaped with a backslash (\). But note that pipes in the extension do not need escaping. For example:

Mar 16 16:26:45 host CEF:0|security|threatmanager|1.0|100|detected a malware \| in message|10|src=10.0.0.192 act=blocked a | dst=12.1.1.1

• If a backslash (\) is used in the prefix or the extension, it has to be escaped with another backslash (\). For example:

Mar 16 16:46:10 host CEF:0|security|threatmanager|1.0|100|detected a malware \\ in packet|10|src=10.0.0.192 act=blocked a \\ dst=1.1.1.1

• If an equal sign (=) is used in the extensions, it has to be escaped with a backslash (\). Equal signs in the prefix need no escaping. For example:

Mar 16 16:46:10 host CEF:0|security|threatmanager|1.0|100|detected a malware \= in packet|10|src=10.0.0.192 act=blocked a \= dst=1.1.1.1

• Multi-line fields can be sent by CEF by encoding the newline character as \n or \r. Note that multiple lines are only allowed in the value part of the extensions. For example:

Mar 16 16:46:10 host CEF:0|security|threatmanager|1.0|100|detected a malware \n in packet|10|src=10.0.0.192 msg=blocked a \n No action needed dst=1.1.1.1

In the previous post you installed the syslog-ng 3.2.2. Now you have to configure our syslog-ng daemon to collect events to database; for this tutorial we choosed a MySQL and Postgres databases. First of all you have to configure the syslog-ng configuration file.

nano /opt/syslog-ng/etc/syslog-ng.conf

Syslog-ng receives log messages from a source. To define a source you should follow the following syntax:

source <identifier> { source-driver(params); source-driver(params); … };

For example you have to define the following source:

source my_source{ tcp ( port ( 614 ) ); };

In syslog-ng log messages are sent to files. The destination syntax is very similar to sources:

destination <identifier> {destination-driver(params); destination-driver(params); … };

You will be normally logging to a file, but you could log to a different destination-driver: pipe, unix socket, TCP-UDP ports, terminals or to specific programs.

destination my_dest{ file(“/var/log/mylog.txt”); };

CREATE DATABASE `syslog` DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;

USE `syslog`;

CREATE TABLE IF NOT EXISTS `logs` (
`id` bigint(20) unsigned NOT NULL auto_increment,
`host` varchar(128) collate utf8_unicode_ci default NULL,
`facility` varchar(10) collate utf8_unicode_ci default NULL,
`priority` varchar(10) collate utf8_unicode_ci default NULL,
`level` varchar(10) collate utf8_unicode_ci default NULL,
`tag` varchar(10) collate utf8_unicode_ci default NULL,
`datetime` datetime default NULL,
`program` varchar(15) collate utf8_unicode_ci default NULL,
`msg` text collate utf8_unicode_ci,
`seq` bigint(20) unsigned NOT NULL default ’0′,
`counter` int(11) NOT NULL default ’1′,
`fo` datetime default NULL,
`lo` datetime default NULL,
PRIMARY KEY (`id`),
KEY `datetime` (`datetime`),
KEY `sequence` (`seq`),
KEY `priority` (`priority`),
KEY `facility` (`facility`),
KEY `program` (`program`),
KEY `host` (`host`) )
ENGINE=MyISAM DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

GRANT SELECT , INSERT , UPDATE , DELETE , CREATE , DROP , INDEX , ALTER ON `syslog` . * TO ‘syslog’@'localhost’;

SET PASSWORD FOR ‘syslog’@'localhost’ = PASSWORD( ‘syslog’ )

Edit syslog-ng config appropriately; add these rows in the destination section (if you want to use Postgres you have to change mysql to pgsql):

sql(type(mysql)
host(“localhost”)
username(“syslog”)
password(“syslog”)
database(“syslog”)
table(“logs”)
columns(“host”, “facility”, “priority”, “level”, “tag”, “datetime”, “program”, “msg”, “seq”)
values(“$HOST_FROM”, “$FACILITY”, “$PRIORITY”, “$LEVEL”, “$TAG”, “$YEAR-$MONTH-$DAY $HOUR:$MIN:$SEC”, “$PROGRAM”, “$MSG”, “$SEQNUM”)
indexes(“host”, “facility”, “priority”, “datetime”, “program”, “seq”));

Syslog-ng connects sources, filters and destinations with log statements. The syntax is:

log { source(src); filter(f_mail); filter(f_info); destination(mailinfo); };

So you have to connect my_source with my_dest:

log { source( my_source ); destination( my_dest ); };

Syslog-ng is an open source implementation of the Syslog protocol for Unix and Unix-like systems. It extends the original syslogd model with content-based filtering, rich filtering capabilities, flexible configuration options and adds important features to syslog, like using TCP for transport. In syslog-ng starting from version 3.0 there is a great option of forward logs directly to database (Postgres, or for that matter to MySQL, Firebird or sqlite database). In comparison with the old way of doing that, namely using a pipe and executing either a wrapper script or mysql client directly, the new way saves a great deal of resources as syslog-ng does not need to start a process every time there is a log message to log. So if you want this features you have to install syslog-ng of version 3.0 or greater with use flag sql enabled. In order to install syslog-ng you have to download the right version from the official site. For our purpose we download the syslog-ng 3.2.2 version (3.2.2/setups/linux-glibc2.3.6-i386).

wget http://www.balabit.com/downloads/files?path=/syslog-ng/sources/3.2.2/setups/linux-glibc2.3.6-i386/syslog-ng-3.2.2-linux-glibc2.3.6-i386.run

Once you downloaded the file you have to grant execute permission to syslog-ng-3.2.2-linux-glibc2.3.6-i386.run.

chmod +x syslog-ng-3.2.2-linux-glibc2.3.6-i386.run

Now you are ready to install the syslog-ng.

./syslog-ng-3.2.2-linux-glibc2.3.6-i386.run

The first screen shows the path where the syslog-ng will be installed; you have to presso “continue”.

The second screen resumes the parameters about your system; press “yes” if the information are corrects.

The third screen suggest user to check if the “/opt/syslog-ng/bin” and “/opt/syslog-ng/sbin” directory are in the search PATH. In order to do so, please add the following line into the shell profile:

PATH=/opt/syslog-ng/bin:$PATH

The fourth step checks if there is old version of syslog-ng installed. If the installer has detected a configuration file from a previous syslog-ng installation, the user can use this old configuration file. We choose “no”.

The installer generates a simple configuration file and asks if user wants to receive log messages from the network. We choose “yes”.

The last step asks user if he wants forward the log messages to a remote server; we choose “skip”.

Congratulation, we installed syslog-ng 3.2.2.

Al fine di velocizzare il processo di installazione di una web application spesso si ricorre a package già “pre-configurati” dove l’unica operazione necessaria è la decompressione dell’archivio. Finché l’applicazione web è “semplice” non si riscontrano problemi ma quando è necessario, per esempio, eseguire comandi nativi unix o comandi di terze parti (generatori di barcode, TSK, tool di scan o simili) iniziano i problemi legati alle librerie. L’errore a cui ci si trova di fronte è il seguente:

/usr/local/bin/fls: /opt/lampp/lib/libcrypto.so.0.9.8: no version information available (required by /usr/local/lib/libafflib.so.0) /usr/local/bin/fls: /opt/lampp/lib/libz.so.1: no version information available (required by /usr/local/lib/libewf.so.1) /usr/local/bin/fls: /opt/lampp/lib/libgcc_s.so.1: version `GCC_4.2.0′ not found (required by /usr/lib/libstdc++.so.6)

Il problema fa riferimento ad un conflitto di versione delle librerie  presenti all’interno della directory /usr/local/lib e /opt/lampp/lib. Da una prima analisi si potrebbe obiettare come l’alternativa più immediata sia quella di installare le singole componenti anziché un pacchetto pre-configurato; d’altro canto questa strada non è la soluzione al problema ma piuttosto un workaround. Durante la fase di debug si è visto come il seguente codice:

putenv(“LD_LIBRARY_PATH=/usr/local/lib”);
echo shell_exec(“/usr/local/bin/fls -V 2>&1″);

risolvesse completamente il problema. Si è scoperto quindi che la natura del problema è da ricercarsi all’interno della variabile d’ambiente LD_LIBRARY_PATH. La variable d’ambiente associata alle librerie può essere modificata in diversi file del sistema operativo (/etc/ld.so.conf, bash.bashrc, .profile, putenv via PHP) ma per essere il meno invasivi possibili è stato scelto di modificare lo script di avvio del servizio LAMPP. Aprendo infatti il file /opt/lampp/lampp, nel dettaglio alla riga 100,  si legge l’istruzione:

export LD_LIBRARY_PATH=/opt/lampp/lib:$LD_LIBRARY_PATH

Modificando l’istruzione precedente in:

export LD_LIBRARY_PATH=/usr/local/lib

vengono risolti tutti i problemi dovuti alle dipendenze richieste durante l’esecuzione del comando d’esempio. Riavviando il servizio /opt/lampp/lampp restart infatti si ottiene l’output corretto.