So I decided to publish the notes I took while setting everything up, and as a little bonus, explain how to install and configure Suricata<\/a> as well (configured in combination with barnyard2 which will pick up local logs and send them to the remote MySQL server).<\/p>\n There are the components that will be installed :<\/p>\n <\/p>\n First, make sure your system is up to date :<\/p>\n \n <\/p>\n \n <\/p>\n (If this shows a different version, then verify that \/usr\/local\/ruby\/bin\/ruby -v is version 1.9.2p0)<\/em><\/p>\n <\/p>\n Edit \/var\/www\/snorby\/config\/database.yml<\/strong> : look for the "snorby" entry and enter the mysql root username & password here :<\/p>\n (don't worry, we'll get rid of the root username\/password later on)<\/p>\n Edit \/var\/www\/snorby\/config\/snorby_config.yml<\/strong> : set the correct path to wkhtmltopdf<\/p>\n <\/p>\n <\/p>\n \n <\/p>\n \n <\/p>\n \n <\/p>\n \n <\/p>\n \n <\/p>\n \n <\/p>\n then run bundle install the rake snorby:setup command again<\/p>\n <\/p>\n <\/p>\n We used the root user \/ password to allow snorby to create the necessary tables. If you prefer to use a mysql user account that has less privileges, then you can add a new mysql user, grant privileges, and edit the snorby configuration again :<\/p>\n \n <\/p>\n You will need to create a useraccount for your (remote) suricata\/snorby sensors too. The procedure is exactly the same as indicated above, but you will have to replace 'localhost' with the IP address of the remote sensor. If the sensor is local, you can use the snorbyuser@localhost<\/a> mysql user account as well.<\/p>\n By default, the mysql server listens on localhost only. Edit \/etc\/mysql\/my.cnf<\/strong> to change the default behaviour :<\/p>\n \n <\/p>\n \n <\/p>\n TCP *:mysql => listening on all interfaces<\/p>\n <\/p>\n \n <\/p>\n <\/p>\n Edit \/etc\/apache2\/mods-available\/passenger.load (or create if it does not exits) :<\/p>\n \n <\/p>\n \n <\/p>\n \n <\/p>\n <\/p>\n Suppose we want the snorby frontend to be reachable using virtualhost snorby.corelan.be :<\/p>\n Create a file "snorby" under \/etc\/apache2\/sites-available<\/strong> :<\/p>\n \n <\/p>\n \n <\/p>\n Make sure snorby.corelan.be points at your local apache2 server, and navigate to that website :<\/p>\n (log in with user snorby@snorby.org<\/a> and password snorby)<\/p>\n If you get an error page instead of the login page :<\/p>\n -> complaining about ezprint.git not being installed, then go to the \/var\/www\/snorby<\/strong> folder and run the following 2 commands :<\/p>\n Wait until the process has finished.<\/p>\n Restart apache2, and then try to access the website again, you should now be able to log on.<\/p>\n <\/p>\n <\/p>\n If you get a message about the "worker" not being started :<\/p>\n Solution : click "Administation", Click "Worker Options" Administration menu and select "Start worker".<\/p>\n Now click on "Worker Options" and start the 2 jobs<\/p>\n If you go back to the main page now, you may see "Currently caching" for a brief moment (depending on the number of events already in the database) :<\/p>\n Tip : if, at any given time, the dashboard continues to show 0 events (or an incorrect number of events in general), but the Events view shows that all entries are inside the database, then you may have to clear the caches and rebuild it from scratch :<\/p>\n Now remove the 2 worker jobs (use the little trash can icon next to each worker job to remove the job)<\/p>\n Recreate the jobs via Worker Options, and the main dashboard should eventually get populated again.<\/p>\n<\/blockquote>\n <\/p>\n Okay, the server is now ready to receive data from local\/remote sensors (Snort, Suricata, \u2026).<\/p>\n <\/p>\n Updating snorby is as easy as running the following commands :<\/p>\n <\/p>\n <\/p>\n Do NOT delete the \/tmp\/barnyard2-1.9 folder yet.<\/p>\n Do NOT remove the \/tmp\/suricata-1.1beta1 folder yet, we need some files from this folder later on.<\/p>\n Try to run suricata :<\/p>\n <\/p>\n If you get the following message :<\/p>\n \n <\/p>\n \n <\/p>\n (note : After copying those files, you can remove the installation folder from \/tmp)<\/em><\/p>\n Edit \/etc\/suricata\/suricata.yaml<\/p>\n Make sure alert output for barnyard2 is enabled (it is enabled by default) :<\/p>\n \n <\/p>\n Next, edit the HOME_NET variable and set it to your local IP or IP subnet<\/p>\n Example :<\/p>\n That's the basic config.<\/p>\n You can use this optional simple script to grab a copy of the git master and update the suricata binaries :<\/p>\n <\/p>\n Get the sample config file from the installation folder :<\/p>\n (note : After copying the file, you can remove the installation folder from \/tmp)<\/em><\/p>\n Edit the conf file and set the following parameters :<\/p>\n (we'll assume you are installing suricata on the same box as the snorby engine)<\/p>\n (obviously the output database configuration must be placed on one line, remove the \/ between the password and dbname.)<\/p>\n If you are installing remote suricate sensors (remote from the mysql server \/ snorby engine point of view), then you will have to configure mysql and grant access to the remote mysqluser, from the IP of the sensor. The "host" entry in the barnyard2.conf file needs to point at the remote mysql server.<\/p>\n Finally, create the log folder for barnyard2 :<\/p>\n This will run barnyard2 in daemon mode. If barnyard2 does not appear to be working, omit the -D parameter and you will be able to see any errors that might prevent barnyard2 from running.<\/p>\n When barnyard2 is running, you should see a new sensor in Snorby. If you don't like the display name of the sensor, you can change the name via Administration Menu - Sensors<\/p>\n When barnyard2 is running, you can launch suricata too :<\/p>\n (change interface accordingly. -D will make suricate run in daemon mode)<\/p>\n <\/p>\n As soon as suricata starts generating alerts, barnyard2 should pick them up, and use the mysql connector to write them into the events table of the snorby database. You should be able to see these new events in the "events" view of Snorby.<\/p>\n In the background (every 30 mins), the snorby worker jobs will pick up the events, process them, add them to the caches table, and show them on the dashboard too.<\/p>\n <\/p>\n If you want to test your setup, then run :<\/p>\n (if lynx was not installed, run apt-get install lynx<\/strong> and try again)<\/p>\n Watch the \/var\/log\/suricata folder. You should see something similar to this :<\/p>\n If the fast.log file, suricata.waldo and unified2.alert files are growing, then the IDS is picking up the test alerts from www.testmyids.com<\/a><\/p>\n <\/p>\n After running this setup for a few days, I noticed that Snorby only appears to be seeing "low severity" events, no matter how hard I try. <\/p>\n Something must be wrong.<\/p>\n I did some tests and it appears that suricata 1.1 might not be able to properly classify events.<\/p>\n I tried with snort, and that seems to work well.<\/p>\n\n
Install dependencies \/ prerequisites for Snorby<\/h3>\n
Packages<\/h4>\n
aptitude update\napt-get update\napt-get upgrade\napt-get dist-upgrade<\/pre>\n
Then install new packages : <\/p>\napt-get install gcc g++ build-essential libssl-dev libreadline5-dev \\\n zlib1g-dev linux-headers-generic libsqlite3-dev libxslt-dev libxml2-dev \\\n imagemagick git-core libmysqlclient-dev mysql-server libmagickwand-dev \\\n default-jre<\/pre>\n
wkhtmlpdf with QT patch<\/h4>\n
cd \/tmp\nwget http:\/\/wkhtmltopdf.googlecode.com\/files\/wkhtmltopdf-0.10.0_rc2-static-i386.tar.bz2\nbunzip2 wkhtmltopdf-0.10.0_rc2-static-i386.tar.bz2\ntar xvf wkhtmltopdf-0.10.0_rc2-static-i386.tar\ncp wkhtmltopdf-i386 \/usr\/bin\/wkhtmltopdf<\/pre>\n
Ruby 1.9.2p0<\/h4>\n
cd \/tmp\nwget http:\/\/ftp.ruby-lang.org\/\/pub\/ruby\/1.9\/ruby-1.9.2-p0.tar.gz\ntar -xvzf ruby-1.9.2-p0.tar.gz\ncd ruby-1.9.2-p0\n.\/configure\nmake && make install\nln -s \/usr\/local\/ruby\/bin\/bundle \/usr\/bin<\/pre>\n
Run "ruby - v" and verify that it returns the correct version : <\/p>\nruby 1.9.2p0 (2010-08-18 revision 29036) [i686-linux]<\/pre>\n
gems<\/h4>\n
gem install thor i18n bundler\ngem install tzinfo builder memcache-client rack rack-test erubis mail text-format\ngem install rack-mount --version=0.4.0\ngem install rails sqlite3-ruby<\/pre>\n
Installing Snorby<\/h3>\n
git clone http:\/\/github.com\/Snorby\/snorby.git \/var\/www\/snorby<\/span><\/pre>\nEdit configuration files :<\/h4>\n
snorby: &snorby\n adapter: mysql\n username: root\n password: <<\/span>enter<\/span> the<\/span> mysql<\/span> root<\/span> password<\/span> here<\/span>><\/span>\n host: localhost<\/pre>\ndevelopment:\n domain: localhost:3000\n wkhtmltopdf: \/usr\/bin\/wkhtmltopdf\n\ntest:\n domain: localhost:3000\n wkhtmltopdf: \/usr\/bin\/wkhtmltopdf\n\nproduction:\n domain: localhost:3000\n wkhtmltopdf: \/usr\/bin\/wkhtmltopdf<\/pre>\n
Run Snorby setup :<\/h4>\n
cd \/var\/www\/snorby\nrake snorby:setup<\/pre>\n
It is very likely that you will get the following error : <\/p>\n(in \/var\/www\/snorby)\nYou have requested:\n activesupport = 3.0.3\n\nThe bundle currently has activesupport locked at 3.0.4.\nTry running `bundle update activesupport`\nTry running `bundle install`.<\/pre>\n
Fix : run the following commands in the \/var\/www\/snorby folder : <\/p>\nbundle update activesupport railties rails\ngem install arel\ngem install ezprint\nbundle install<\/pre>\n
Run the setup again : <\/p>\ncd \/var\/www\/snorby\nrake snorby:setup<\/pre>\n
If all goes well, the snorby database should get created\/populated now. Since we used the mysql root username\/password in the database.yml configuration file, the necessary database and tables should be created successfully. <\/p>\nroot@server:\/var\/www\/snorby# rake snorby:setup\n(in \/var\/www\/snorby)\n<...long key....>\n[datamapper] Created database 'snorby'\n[datamapper] Finished auto_upgrade! for :default repository 'snorby'<\/pre>\n
If you get an error about ezprint: <\/p>\n(in \/var\/www\/snorby)\nrake aborted!\nhttp:\/\/github.com\/mephux\/ezprint.git (at rails3) is not checked out. \\ <\/span>\n Please run `bundle install`\n\/var\/www\/snorby\/Rakefile:4\n(See full trace by running task with --trace)<\/pre>\n
Solution : run this from \/var\/www\/snorby<\/strong> <\/p>\nbundle pack\nbundle install --path vender\/cache<\/pre>\n
Configure mysql<\/h4>\n
mysql -u root -p\n\ncreat user 'snorbyuser'@'localhost' IDENTIFIED BY 'some_pass';\ngrant all privileges on snorby.* to 'snorbyuser'@'localhost' with grant option;\nflush privileges;<\/pre>\n
Now edit \/var\/www\/snorby\/config\/database.yml again and replace the username and password with the newly created user <\/p>\nsnorby: &snorby\n adapter: mysql\n username: snorbyuser\n password: some_pass\n host: localhost<\/pre>\n
# Instead of skip-networking the default is now to listen only on\n# localhost which is more compatible and is not less secure.\nbind-address = 127.0.0.1<\/pre>\n
Comment the bind-address statement (add a # in front of the line) and restart mysql <\/p>\nservice mysql restart<\/pre>\n
Verify that the server is now listening on all ip addresses : <\/p>\nroot@server:\/# lsof -i | grep mysqld\nmysqld 21309 mysql 10u IPv4 16405476 0t0 TCP *:mysql (LISTEN)<\/pre>\n
Apache2 & Passenger<\/h3>\n
Install packages & dependencies<\/h4>\n
apt-get install apache2 apache2-prefork-dev libapr1-dev libaprutil1-dev libopenssl-ruby\napt-get install libcurl4-openssl-dev<\/pre>\n
Start apache2 and make sure the default webpage loads <\/p>\nservice apache2 start<\/pre>\n
Install passenger<\/h4>\n
gem install --no-ri --no-rdoc --version 3.0.3 passenger<\/pre>\n
Install passenger module for apache2<\/h4>\n
\/usr\/local\/ruby\/lib\/ruby\/gems\/1.9.1\/gems\/passenger-3.0.3\/bin\/passenger-install-apache2-module -a<\/pre>\n
LoadModule passenger_module \/usr\/local\/ruby\/lib\/ruby\/gems\/1.9.1\/gems\/passenger-3.0.3\/ext\/apache2\/mod_passenger.so<\/pre>\n
Edit \/etc\/apache2\/mods-available\/passenger.conf : <\/p>\n<IfModule mod_passenger.c>\n PassengerRoot \/usr\/local\/ruby\/lib\/ruby\/gems\/1.9.1\/gems\/passenger-3.0.3\n PassengerRuby \/usr\/local\/ruby\/bin\/ruby\n<\/IfModule><\/pre>\n
Enable the module (and some other modules you might need) : <\/p>\na2enmod passenger\na2enmod rewrite\na2enmod ssl<\/pre>\n
Set file\/folder permissions on the snorby folder : <\/p>\nchown www-data:www-data \/var\/www\/snorby -R<\/pre>\n
Integrate Snorby with Apache2<\/h4>\n
<VirtualHost *:80>\n ServerAdmin webmaster@localhost\n ServerName snorby.corelan.be\n DocumentRoot \/var\/www\/snorby\/public\n\n <Directory "\/var\/www\/snorby\/public<\/span>">\n AllowOverride all\n Order deny,allow\n Allow from all\n Options -MultiViews\n <\/Directory>\n\n<\/VirtualHost><\/pre>\n
Enable the new website : <\/p>\nln -s \/etc\/apache2\/sites-available\/snorby \/etc\/apache2\/sites-enabled\/snorby<\/pre>\n
Restart apache2 : <\/p>\nservice apache2 restart<\/pre>\n
![\"image\"](\"http:\/\/www.corelan.be\/wp-content\/uploads\/2011\/02\/image_thumb.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"http:\/\/www.corelan.be\/wp-content\/uploads\/2011\/02\/image_thumb5.png\" "\\"image\\"")
<\/a><\/p>\nbundle pack\nbundle install --path vender\/cache<\/pre>\n
![\"image\"](\"http:\/\/www.corelan.be\/wp-content\/uploads\/2011\/02\/image_thumb6.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"http:\/\/www.corelan.be\/wp-content\/uploads\/2011\/02\/image_thumb1.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"http:\/\/www.corelan.be\/wp-content\/uploads\/2011\/02\/image_thumb2.png\" "\\"image\\"")
<\/a><\/p>\n![\"image\"](\"http:\/\/www.corelan.be\/wp-content\/uploads\/2011\/02\/image_thumb3.png\" "\\"image\\"")
<\/a><\/p>\n\n
mysql -u root -p\n\nuse snorby;\ntruncate table caches;\nexit<\/span><\/pre>\n![\"image\"](\"http:\/\/www.corelan.be\/wp-content\/uploads\/2011\/02\/image_thumb4.png\" "\\"image\\"")
<\/a><\/p>\nUpdating Snorby<\/h3>\n
cd \/var\/www\/snorby\ngit pull origin master\nrake snorby:update<\/pre>\n
Installing Suricata & Barnyard2<\/h3>\n
Dependencies<\/h4>\n
apt-get install libpcre3 libpcre3-dbg libpcre3-dev \\\n build-essential autoconf automake libtool \\\n libpcap-dev libnet1-dev mysql-client libmysqlclient16-dev<\/pre>\n
Set up yaml :<\/h4>\n
yaml :\ncd \/tmp\nwget http:\/\/pyyaml.org\/download\/libyaml\/yaml-0.1.3.tar.gz<\/span>\ntar xvfz yaml-0.1.3.tar.gz\ncd yaml-0.1.3\n.\/configure && make && make install<\/pre>\nInstall barnyard2 :<\/h4>\n
cd \/tmp\nwget http:\/\/www.securixlive.com\/download\/barnyard2\/barnyard2-1.9.tar.gz<\/span>\ntar xvfz barnyard2-1.9.tar.gz\ncd barnyard2-1.9\n.\/configure --with-mysql && make && make install<\/pre>\nInstall suricata:<\/h4>\n
cd \/tmp\nwget http:\/\/www.openinfosecfoundation.org\/download\/suricata-1.1beta1.tar.gz<\/span>\ntar xvfz suricata-1.1beta1.tar.gz\ncd suricata-1.1beta1\nmkdir \/var\/log\/suricata\n.\/configure && make && make install<\/pre>\nsuricata<\/pre>\n
suricata: error while loading shared libraries: libhtp-0.2.so.1: cannot open shared object file: No such file or directory<\/pre>\n
then add "\/usr\/local\/lib" to \/etc\/ld.so.conf and run ldconfig. <\/p>\nroot@server:\/# cat \/etc\/ld.so.conf\ninclude \/etc\/ld.so.conf.d\/*.conf\n\/usr\/local\/lib\nroot@server:\/# ldconfig<\/span><\/pre>\n
Run "suricata" again : <\/p>\nsuricata\n[14005] 27\/2\/2011 -- 22:08:28 - (suricata.c:440) <Info> (main) -- This is Suricata version 1.1beta1\n[14005] 27\/2\/2011 -- 22:08:28 - (util-cpu.c:171) <Info> (UtilCpuPrintSummary) -- CPUs\/cores online: 2\n[14005] 27\/2\/2011 -- 22:08:28 - (suricata.c:765) <Error> (main) -- [ERRCODE: SC_ERR_OPENING_FILE(40)]\n - Configuration file has not been provided\n\nSuricata 1.1beta1\nUSAGE: suricata\n\n -c <path> : path to configuration file\n -i <dev or ip> : run in pcap live mode\n -r <path> : run in pcap file\/offline mode\n -s <path> : path to signature file (optional)\n -l <dir> : default log directory\n -D : run as daemon\n --engine-analysis : print reports on analysis of different sections in the engine and exit.\n Please have a look at the conf parameter engine-analysis on what reports\n can be printed\n --pidfile <file> : write pid to this file (only for daemon mode)\n --init-errors-fatal : enable fatal failure on signature init error\n --dump-config : show the running configuration\n --pcap-buffer-size : size of the pcap buffer value from 0 - 2147483647\n --user <user> : run suricata as this user after init\n --group <group> : run suricata as this group after init\n --erf-in <path> : process an ERF file\n\nTo run the engine with default configuration on interface eth0\nwith signature file "signatures.rules<\/span>", run the command as:\n\nsuricata -c suricata.yaml -s signatures.rules -i eth0<\/pre>\nGet suricata rules (emerging-threats)<\/h4>\n
mkdir \/etc\/suricata\ncd \/etc\/suricata\nwget http:\/\/rules.emergingthreats.net\/open\/suricata\/emerging.rules.tar.gz<\/span>\ntar xvfz emerging.rules.tar.gz<\/pre>\nConfigure suricata :<\/h4>\n
cd \/tmp\/suricata-1.1beta1\ncp suricata.yaml \/etc\/suricata\/\ncp classification.config \/etc\/suricata\/\ncp reference.config \/etc\/suricata\/<\/pre>\n
# alert output for use with Barnyard2\n - unified2-alert:\n enabled: yes\n filename: unified2.alert\n\n # Limit in MB.\n #limit: 32<\/pre>\n
Scroll down until you reach "default-rule-path:" and enable\/put the emerging-threat rules files that are relevant to your system under "rule-files:". (You can find the list with rules under \/etc\/suricata\/rules). Example : <\/p>\ndefault-rule-path: \/etc\/suricata\/rules\/\nrule-files:\n - emerging-attack_response.rules\n - emerging-dos.rules\n - emerging-exploit.rules\n - emerging-games.rules\n - emerging-inappropriate.rules\n - emerging-malware.rules\n - emerging-p2p.rules\n - emerging-policy.rules\n - emerging-scada.rules\n - emerging-smtp.rules\n - emerging-virus.rules\n - emerging-voip.rules\n - emerging-web_client.rules\n - emerging-web_server.rules\n - emerging-web_specific_apps.rules\n - emerging-worm.rules\n - emerging-user_agents.rules\n - emerging-current_events.rules<\/pre>\n
HOME_NET: "[192.168.0.0\/24]<\/span>"<\/pre>\nKeeping suricata up to date<\/h4>\n
#!\/bin\/bash\ncd \/tmp\nrm -rf \/tmp\/suricata\nmkdir suricata\ncd suricata\n\/usr\/bin\/git clone git:\/\/phalanx.openinfosecfoundation.org\/oisf.git\ncd oisf\n.\/autogen.sh\n.\/configure && make && make install<\/pre>\n
Configure barnyard2 :<\/h4>\n
cp \/tmp\/barnyard2-1.9\/etc\/barnyard2.conf \/etc\/suricata\/<\/pre>\n
config reference_file: \/etc\/suricata\/reference.config\nconfig classification_file: \/etc\/suricata\/classification.config\nconfig gen_file: \/etc\/suricata\/rules\/gen-msg.map\nconfig sid_file: \/etc\/suricata\/rules\/sid-msg.map\n\noutput database: log, mysql, user=snorbyuser password=some_pass \/\n dbname=snorby host=localhost sensor_name=sensor1<\/pre>\n
mkdir \/var\/log\/barnyard2<\/pre>\n
Run barnyard2 :<\/h3>\n
barnyard2 -c \/etc\/suricata\/barnyard2.conf -d \/var\/log\/suricata -f unified2.alert -w \/var\/log\/suricata\/suricata.waldo -D<\/pre>\n
Run suricata :<\/h3>\n
suricata -c \/etc\/suricata\/suricata.yaml -i eth0 -D<\/pre>\n
Test IDS<\/h3>\n
lynx www.testmyids.com<\/pre>\n
root@server:\/var\/log\/suricata# ls -al\ntotal 88\ndrwxr-xr-x 2 root root 4096 2011-02-28 05:38 .\ndrwxr-xr-x 18 root root 4096 2011-02-28 05:30 ..\n-rw-r----- 1 root root 194 2011-02-28 05:37 fast.log\n-rw-r----- 1 root root 0 2011-02-28 05:35 http.log\n-rw-r--r-- 1 root root 66873 2011-02-28 05:39 stats.log\n-rw------- 1 root root 2056 2011-02-28 05:38 suricata.waldo\n-rw-r----- 1 root root 0 2011-02-28 05:34 unified2.alert.1298867650\n-rw-r----- 1 root root 60 2011-02-28 05:37 unified2.alert.1298867720<\/pre>\n
Addendum (march 2nd 2011)<\/h3>\n