Corelan Team » 001_Security » Papers » Cheat sheet : Installing Snorby 2.2 with Apache2 and Suricata with Barnyard2 on Ubuntu 10.x
17,427 viewsThis page as PDF (Login first !)
Cheat sheet : Installing Snorby 2.2 with Apache2 and Suricata with Barnyard2 on Ubuntu 10.x
After spending a few hours fighting a battle against Snorby and Apache2 + Passenger, I finally managed to get it to run properly on my Ubunty 10.x box (32bit). Looking back, I figured I might not be the only one who is having issues with this.
So I decided to publish the notes I took while setting everything up, and as a little bonus, explain how to install and configure Suricata as well (configured in combination with barnyard2 which will pick up local logs and send them to the remote MySQL server).
There are the components that will be installed :
- Snorby 2.x (latest revision from git)
- MySql 5
- Ruby 1.9.2p0
- Passenger 3
- Barnyard 2
- Suricata 1.1beta1 with emerging-threat ruleset
Install dependencies / prerequisites for Snorby
First, make sure your system is up to date :
aptitude update apt-get update apt-get upgrade apt-get dist-upgrade
Then install new packages :
apt-get install gcc g++ build-essential libssl-dev libreadline5-dev \ zlib1g-dev linux-headers-generic libsqlite3-dev libxslt-dev libxml2-dev \ imagemagick git-core libmysqlclient-dev mysql-server libmagickwand-dev \ default-jre
wkhtmlpdf with QT patch
cd /tmp wget http://wkhtmltopdf.googlecode.com/files/wkhtmltopdf-0.10.0_rc2-static-i386.tar.bz2 bunzip2 wkhtmltopdf-0.10.0_rc2-static-i386.tar.bz2 tar xvf wkhtmltopdf-0.10.0_rc2-static-i386.tar cp wkhtmltopdf-i386 /usr/bin/wkhtmltopdf
cd /tmp wget http://ftp.ruby-lang.org//pub/ruby/1.9/ruby-1.9.2-p0.tar.gz tar -xvzf ruby-1.9.2-p0.tar.gz cd ruby-1.9.2-p0 ./configure make && make install ln -s /usr/local/ruby/bin/bundle /usr/bin
Run "ruby – v" and verify that it returns the correct version :
ruby 1.9.2p0 (2010-08-18 revision 29036) [i686-linux]
(If this shows a different version, then verify that /usr/local/ruby/bin/ruby -v is version 1.9.2p0)
gem install thor i18n bundler gem install tzinfo builder memcache-client rack rack-test erubis mail text-format gem install rack-mount --version=0.4.0 gem install rails sqlite3-ruby
git clone http://github.com/Snorby/snorby.git /var/www/snorby
Edit configuration files :
Edit /var/www/snorby/config/database.yml : look for the "snorby" entry and enter the mysql root username & password here :
snorby: &snorby adapter: mysql username: root password: <enter the mysql root password here> host: localhost
(don’t worry, we’ll get rid of the root username/password later on)
Edit /var/www/snorby/config/snorby_config.yml : set the correct path to wkhtmltopdf
development: domain: localhost:3000 wkhtmltopdf: /usr/bin/wkhtmltopdf test: domain: localhost:3000 wkhtmltopdf: /usr/bin/wkhtmltopdf production: domain: localhost:3000 wkhtmltopdf: /usr/bin/wkhtmltopdf
Run Snorby setup :
cd /var/www/snorby rake snorby:setup
It is very likely that you will get the following error :
(in /var/www/snorby) You have requested: activesupport = 3.0.3 The bundle currently has activesupport locked at 3.0.4. Try running `bundle update activesupport` Try running `bundle install`.
Fix : run the following commands in the /var/www/snorby folder :
bundle update activesupport railties rails gem install arel gem install ezprint bundle install
Run the setup again :
cd /var/www/snorby rake snorby:setup
If all goes well, the snorby database should get created/populated now. Since we used the mysql root username/password in the database.yml configuration file, the necessary database and tables should be created successfully.
root@server:/var/www/snorby# rake snorby:setup (in /var/www/snorby) <...long key....> [datamapper] Created database 'snorby' [datamapper] Finished auto_upgrade! for :default repository 'snorby'
If you get an error about ezprint:
(in /var/www/snorby) rake aborted! http://github.com/mephux/ezprint.git (at rails3) is not checked out. \ Please run `bundle install` /var/www/snorby/Rakefile:4 (See full trace by running task with --trace)
Solution : run this from /var/www/snorby
bundle pack bundle install --path vender/cache
then run bundle install the rake snorby:setup command again
We used the root user / password to allow snorby to create the necessary tables. If you prefer to use a mysql user account that has less privileges, then you can add a new mysql user, grant privileges, and edit the snorby configuration again :
mysql -u root -p creat user 'snorbyuser'@'localhost' IDENTIFIED BY 'some_pass'; grant all privileges on snorby.* to 'snorbyuser'@'localhost' with grant option; flush privileges;
Now edit /var/www/snorby/config/database.yml again and replace the username and password with the newly created user
snorby: &snorby adapter: mysql username: snorbyuser password: some_pass host: localhost
You will need to create a useraccount for your (remote) suricata/snorby sensors too. The procedure is exactly the same as indicated above, but you will have to replace ‘localhost’ with the IP address of the remote sensor. If the sensor is local, you can use the snorbyuser@localhost mysql user account as well.
By default, the mysql server listens on localhost only. Edit /etc/mysql/my.cnf to change the default behaviour :
# Instead of skip-networking the default is now to listen only on # localhost which is more compatible and is not less secure. bind-address = 127.0.0.1
Comment the bind-address statement (add a # in front of the line) and restart mysql
service mysql restart
Verify that the server is now listening on all ip addresses :
root@server:/# lsof -i | grep mysqld mysqld 21309 mysql 10u IPv4 16405476 0t0 TCP *:mysql (LISTEN)
TCP *:mysql => listening on all interfaces
Apache2 & Passenger
Install packages & dependencies
apt-get install apache2 apache2-prefork-dev libapr1-dev libaprutil1-dev libopenssl-ruby apt-get install libcurl4-openssl-dev
Start apache2 and make sure the default webpage loads
service apache2 start
gem install --no-ri --no-rdoc --version 3.0.3 passenger
Install passenger module for apache2
Edit /etc/apache2/mods-available/passenger.load (or create if it does not exits) :
LoadModule passenger_module /usr/local/ruby/lib/ruby/gems/1.9.1/gems/passenger-3.0.3/ext/apache2/mod_passenger.so
Edit /etc/apache2/mods-available/passenger.conf :
<IfModule mod_passenger.c> PassengerRoot /usr/local/ruby/lib/ruby/gems/1.9.1/gems/passenger-3.0.3 PassengerRuby /usr/local/ruby/bin/ruby </IfModule>
Enable the module (and some other modules you might need) :
a2enmod passenger a2enmod rewrite a2enmod ssl
Set file/folder permissions on the snorby folder :
chown www-data:www-data /var/www/snorby -R
Integrate Snorby with Apache2
Suppose we want the snorby frontend to be reachable using virtualhost snorby.corelan.be :
Create a file "snorby" under /etc/apache2/sites-available :
<VirtualHost *:80> ServerAdmin webmaster@localhost ServerName snorby.corelan.be DocumentRoot /var/www/snorby/public <Directory "/var/www/snorby/public"> AllowOverride all Order deny,allow Allow from all Options -MultiViews </Directory> </VirtualHost>
Enable the new website :
ln -s /etc/apache2/sites-available/snorby /etc/apache2/sites-enabled/snorby
Restart apache2 :
service apache2 restart
Make sure snorby.corelan.be points at your local apache2 server, and navigate to that website :
(log in with user email@example.com and password snorby)
If you get an error page instead of the login page :
-> complaining about ezprint.git not being installed, then go to the /var/www/snorby folder and run the following 2 commands :
bundle pack bundle install --path vender/cache
Wait until the process has finished.
Restart apache2, and then try to access the website again, you should now be able to log on.
If you get a message about the "worker" not being started :
Solution : click "Administation", Click "Worker Options" Administration menu and select "Start worker".
Now click on "Worker Options" and start the 2 jobs
If you go back to the main page now, you may see "Currently caching" for a brief moment (depending on the number of events already in the database) :
Tip : if, at any given time, the dashboard continues to show 0 events (or an incorrect number of events in general), but the Events view shows that all entries are inside the database, then you may have to clear the caches and rebuild it from scratch :mysql -u root -p use snorby; truncate table caches; exit
Now remove the 2 worker jobs (use the little trash can icon next to each worker job to remove the job)
Recreate the jobs via Worker Options, and the main dashboard should eventually get populated again.
Okay, the server is now ready to receive data from local/remote sensors (Snort, Suricata, …).
Updating snorby is as easy as running the following commands :
cd /var/www/snorby git pull origin master rake snorby:update
Installing Suricata & Barnyard2
apt-get install libpcre3 libpcre3-dbg libpcre3-dev \ build-essential autoconf automake libtool \ libpcap-dev libnet1-dev mysql-client libmysqlclient16-dev
Set up yaml :
yaml : cd /tmp wget http://pyyaml.org/download/libyaml/yaml-0.1.3.tar.gz tar xvfz yaml-0.1.3.tar.gz cd yaml-0.1.3 ./configure && make && make install
Install barnyard2 :
cd /tmp wget http://www.securixlive.com/download/barnyard2/barnyard2-1.9.tar.gz tar xvfz barnyard2-1.9.tar.gz cd barnyard2-1.9 ./configure --with-mysql && make && make install
Do NOT delete the /tmp/barnyard2-1.9 folder yet.
cd /tmp wget http://www.openinfosecfoundation.org/download/suricata-1.1beta1.tar.gz tar xvfz suricata-1.1beta1.tar.gz cd suricata-1.1beta1 mkdir /var/log/suricata ./configure && make && make install
Do NOT remove the /tmp/suricata-1.1beta1 folder yet, we need some files from this folder later on.
Try to run suricata :
If you get the following message :
suricata: error while loading shared libraries: libhtp-0.2.so.1: cannot open shared object file: No such file or directory
then add "/usr/local/lib" to /etc/ld.so.conf and run ldconfig.
root@server:/# cat /etc/ld.so.conf include /etc/ld.so.conf.d/*.conf /usr/local/lib root@server:/# ldconfig
Run "suricata" again :
suricata  27/2/2011 -- 22:08:28 - (suricata.c:440) <Info> (main) -- This is Suricata version 1.1beta1  27/2/2011 -- 22:08:28 - (util-cpu.c:171) <Info> (UtilCpuPrintSummary) -- CPUs/cores online: 2  27/2/2011 -- 22:08:28 - (suricata.c:765) <Error> (main) -- [ERRCODE: SC_ERR_OPENING_FILE(40)] - Configuration file has not been provided Suricata 1.1beta1 USAGE: suricata -c <path> : path to configuration file -i <dev or ip> : run in pcap live mode -r <path> : run in pcap file/offline mode -s <path> : path to signature file (optional) -l <dir> : default log directory -D : run as daemon --engine-analysis : print reports on analysis of different sections in the engine and exit. Please have a look at the conf parameter engine-analysis on what reports can be printed --pidfile <file> : write pid to this file (only for daemon mode) --init-errors-fatal : enable fatal failure on signature init error --dump-config : show the running configuration --pcap-buffer-size : size of the pcap buffer value from 0 - 2147483647 --user <user> : run suricata as this user after init --group <group> : run suricata as this group after init --erf-in <path> : process an ERF file To run the engine with default configuration on interface eth0 with signature file "signatures.rules", run the command as: suricata -c suricata.yaml -s signatures.rules -i eth0
Get suricata rules (emerging-threats)
mkdir /etc/suricata cd /etc/suricata wget http://rules.emergingthreats.net/open/suricata/emerging.rules.tar.gz tar xvfz emerging.rules.tar.gz
Configure suricata :
cd /tmp/suricata-1.1beta1 cp suricata.yaml /etc/suricata/ cp classification.config /etc/suricata/ cp reference.config /etc/suricata/
(note : After copying those files, you can remove the installation folder from /tmp)
Make sure alert output for barnyard2 is enabled (it is enabled by default) :
# alert output for use with Barnyard2 - unified2-alert: enabled: yes filename: unified2.alert # Limit in MB. #limit: 32
Scroll down until you reach "default-rule-path:" and enable/put the emerging-threat rules files that are relevant to your system under "rule-files:". (You can find the list with rules under /etc/suricata/rules). Example :
default-rule-path: /etc/suricata/rules/ rule-files: - emerging-attack_response.rules - emerging-dos.rules - emerging-exploit.rules - emerging-games.rules - emerging-inappropriate.rules - emerging-malware.rules - emerging-p2p.rules - emerging-policy.rules - emerging-scada.rules - emerging-smtp.rules - emerging-virus.rules - emerging-voip.rules - emerging-web_client.rules - emerging-web_server.rules - emerging-web_specific_apps.rules - emerging-worm.rules - emerging-user_agents.rules - emerging-current_events.rules
Next, edit the HOME_NET variable and set it to your local IP or IP subnet
That’s the basic config.
Keeping suricata up to date
You can use this optional simple script to grab a copy of the git master and update the suricata binaries :
#!/bin/bash cd /tmp rm -rf /tmp/suricata mkdir suricata cd suricata /usr/bin/git clone git://phalanx.openinfosecfoundation.org/oisf.git cd oisf ./autogen.sh ./configure && make && make install
Configure barnyard2 :
Get the sample config file from the installation folder :
cp /tmp/barnyard2-1.9/etc/barnyard2.conf /etc/suricata/
(note : After copying the file, you can remove the installation folder from /tmp)
Edit the conf file and set the following parameters :
(we’ll assume you are installing suricata on the same box as the snorby engine)
config reference_file: /etc/suricata/reference.config config classification_file: /etc/suricata/classification.config config gen_file: /etc/suricata/rules/gen-msg.map config sid_file: /etc/suricata/rules/sid-msg.map output database: log, mysql, user=snorbyuser password=some_pass / dbname=snorby host=localhost sensor_name=sensor1
(obviously the output database configuration must be placed on one line, remove the / between the password and dbname.)
If you are installing remote suricate sensors (remote from the mysql server / snorby engine point of view), then you will have to configure mysql and grant access to the remote mysqluser, from the IP of the sensor. The "host" entry in the barnyard2.conf file needs to point at the remote mysql server.
Finally, create the log folder for barnyard2 :
Run barnyard2 :
barnyard2 -c /etc/suricata/barnyard2.conf -d /var/log/suricata -f unified2.alert -w /var/log/suricata/suricata.waldo -D
This will run barnyard2 in daemon mode. If barnyard2 does not appear to be working, omit the -D parameter and you will be able to see any errors that might prevent barnyard2 from running.
When barnyard2 is running, you should see a new sensor in Snorby. If you don’t like the display name of the sensor, you can change the name via Administration Menu – Sensors
When barnyard2 is running, you can launch suricata too :
Run suricata :
suricata -c /etc/suricata/suricata.yaml -i eth0 -D
(change interface accordingly. -D will make suricate run in daemon mode)
As soon as suricata starts generating alerts, barnyard2 should pick them up, and use the mysql connector to write them into the events table of the snorby database. You should be able to see these new events in the "events" view of Snorby.
In the background (every 30 mins), the snorby worker jobs will pick up the events, process them, add them to the caches table, and show them on the dashboard too.
If you want to test your setup, then run :
(if lynx was not installed, run apt-get install lynx and try again)
Watch the /var/log/suricata folder. You should see something similar to this :
root@server:/var/log/suricata# ls -al total 88 drwxr-xr-x 2 root root 4096 2011-02-28 05:38 . drwxr-xr-x 18 root root 4096 2011-02-28 05:30 .. -rw-r----- 1 root root 194 2011-02-28 05:37 fast.log -rw-r----- 1 root root 0 2011-02-28 05:35 http.log -rw-r--r-- 1 root root 66873 2011-02-28 05:39 stats.log -rw------- 1 root root 2056 2011-02-28 05:38 suricata.waldo -rw-r----- 1 root root 0 2011-02-28 05:34 unified2.alert.1298867650 -rw-r----- 1 root root 60 2011-02-28 05:37 unified2.alert.1298867720
If the fast.log file, suricata.waldo and unified2.alert files are growing, then the IDS is picking up the test alerts from www.testmyids.com
Addendum (march 2nd 2011)
After running this setup for a few days, I noticed that Snorby only appears to be seeing "low severity" events, no matter how hard I try.
Something must be wrong.
I did some tests and it appears that suricata 1.1 might not be able to properly classify events.
I tried with snort, and that seems to work well.
Update : I filed a bug report with suricata and it looks like they have fixed the issue. You can use the suricata "update" script to grab the latest version.
Follow the installation guide at http://www.snort.org/assets/158/snortinstallguide2904.pdf with the exception of creating a "snort" database. Make sure to set up barnyard2 logging and tell it to use "snorby" instead of the "snort" database.
In short, the easiest way to get snort to run on ubuntu 10.x is to run apt-get install snort (or apt-get install snort-mysql if you want to have snort log events directly into the MySql database. This is not the recommended configuration and barnyard2 will be able to handle MySQL downtime)
© 2011, Corelan Team (corelanc0d3r). All rights reserved.