Scaling
Introduction
Our maximum throughput on single server (24 cores Xeon, 10Gbit NIC card) is around 20 000 calls. But VoIPmonitor can work in cluster mode where remote sensors writes to one central database with central GUI server. Usual 4 core Xeon server (E3-1220) is able to handle up to 4000 simultaneous calls and probably more.
VoIPmonitor is able to use all available CPU cores but there are several bottlenecks which you should consider before deploying and configuring VoIPmonitor. We do free full presale support in case you need help to deploy our solution.
Basically there are three types of bottlenecks - CPU, I/O throughput and MySQL performance. The sniffer is multithreaded application but certain tasks cannot be split to more threads. Main thread is reading packets from kernel - this is the top most consuming thread and it depends on CPU type and kernel version (and number of packets per second). Below 1000 concurrent calls you do not need to be worried about CPU on usual CPU (Xeon, i5). More details about CPU bottleneck see following chapter CPU bound.
I/O bottleneck is most common problem for voipmonitor and it depends if you store to local mysql database along with storing pcap files on the same server and the same storage. Differences between I/O hardware and file system are so significant that there is no general recommendation and the problem can be for 100 concurrent calls. See next chapter I/O throughput.
CPU bound
Reading packets
Main thread which reads packets from kernel cannot be split into more threads which limits number of concurrent calls for the whole server. CPU used for this thread is equivalent to running "tcpdump -i ethX -w /dev/null" which you can use as a test if your server is able to handle your traffic. We have tested sniffer on countless type of servers and basically the limit is somewhere around at 800Mbit for usual 1Gbit card on newer Xeon CPU and kernel versions >= 2.6.32. To get higher throughput special drivers or hardware is needed.
On following picture you can see how packets are proccessed from ethernet card to Kernel space to ethernet driver which queues packets to ring buffer. Ring buffer (available since kernel 2.6.32 and libpcap > 1.0) is read by libpcap functions in voipmonitor to its own voipmonitor buffer. Kernel ring buffer is circular buffer directly in kernel which reads packets from ethernet card and overwrites the oldest one if not read in time. Ring buffer can be large at maximum 2GB. If voipmonitor is blocked by CPU or I/O the ring buffer starts filling up to its maximum set size (voipmonitor.conf:ringbuffer=XXX) and in this case dropping packets occurs which is logged into the syslog. VoIPmonitor sniffer reads packets from ring buffer in one main thread and writing packets to voipmonitor buffer with maximum size 4GB (voipmonitor.conf:vmbuffer=XXXX). Which means that maximum buffer size is 6GB RAM which can cover I/O low throughput until all buffers are filled. Thus if you set both buffers to its maximum values and your I/O is not able to handle save all packets to disk (if saving is enabled) you can see dropping packets after 6GB of packets fills buffers. VoIPmonitor buffer is read by another thread to one or more queues which depends on how much number of CPU is available (voipmonitor.conf:rtpthreads=X). Those queues processes RTP packets in parallel. Jitterbuffer simulater uses the most CPU and you can disable all three type of jitterbuffers if your server is not able to handle it all (jitterbuffer_f1, jitterbuffer_f2, jitterbuffer_adapt). If you need to disable one of the jitterbuffer keep jitterbuffer_f2 enabled which is the most usefull.
Good tool for measuring CPU is http://htop.sourceforge.net/
Software driver alternatives
If your traffic is to much for your current hardware you can try PF_RING feature.
- PF_RING http://www.ntop.org/products/pf_ring/
- Direct NIC Access http://www.ntop.org/products/pf_ring/dna/
We tried DNA driver for stock 1Gbit Intel card which reduces 100% CPU load to 20% but we still saw occasional packet loss on the card - although the loss was minimal.
Hardware NIC cards
We have tested 10Gbit cards from Napatech which can handle at least 20000 concurrent calls with 0% CPU for main thread. Those cards are very expensive but the performance is worth it.
I/O bottleneck
For storing up to 100 simultaneous calls you do not need to be worried about I/O performance much. For storing up to 500 calls your disk must have enabled write cache (some raid controllers are not set well for random write scenarios or has write cache disabled at all). For up to 1000 calls you can use ordinary SATA 7.2kRPM disks with NCQ enabled - like Western digital RE4 edition (RE4 is important as it implements good NCQ) and we use it for installations for saving full SIP+RTP up to 1000 simultaneous calls. If you have more than 1000 simultaneous calls you can still use usual SATA disk but using cachedir feature (see below) or you need to look for some hardware raid and test the performance before you buy! Performance of such raids varies a lot and there is no general recommendation or working solutions which we can provide as a reference.
SSD disks are not recommended for pcap storing because of its low durability.
VoIPmonitor sniffer produces the worst case scenario for spin disks - random write. The situation gets worse in case of ext3/ext4 file systems which uses journal and writes meta data enabled by default thus adding more I/O writes. But ext4 can be tweaked to get maximum performance. We are recommending to use dedicated disk and format it with special ext4 switches which turns off journal and other features which removes some safety in favor of performance. If you cannot use dedicated disk for storing pcap files use dedicated partition formatted with special tweaks (see below).
The fastest filesystem for voipmonitor spool directory is EXT4 with turned off journaling and other tweaks. Assuming your partition is /dev/sda2:
mke2fs -t ext4 -O ^has_journal /dev/sda2 tune2fs -O ^has_journal /dev/sda2 tune2fs -o journal_data_writeback /dev/sda2 #add following line to /etc/fstab /dev/sda2 /var/spool/voipmonitor ext4 errors=remount-ro,noatime,nodiratime,data=writeback,barrier=0 0 0
In case your disk is still not able to handle traffic you can enable cachedir feature (voipmonitor.conf:cachedir) which stores all files into fast storage which can handle random write - for example RAM disk located at /dev/shm (every linux distribution have enabled this for up to 50% of memory). After the file is closed (call ends) voipmonitor automatically move the file from this storage to spooldir directory which is located on slower storage in guaranteed serial order which eliminates bad random write problem. This also allows to use network shares which is usually too slow to use it for writing directly to it by voipmonitor sniffer.
MySQL performance
Write performance
Write performance depends a lot if a storage is also used for pcap storing (thus sharing I/O with voipmonitor) and on how mysql handles writes (innodb_flush_log_at_trx_commit parameter - see below). Since sniffer version 6 MySQL tables uses compression which doubles write and read performance almost with no trade cost on CPU.
innodb_flush_log_at_trx_commit
Default value of 1 will mean each update transaction commit (or each statement outside of transaction) will need to flush log to the disk which is rather expensive, especially if you do not have Battery backed up cache. Many applications are OK with value 2 which means do not flush log to the disk but only flush it to OS cache. The log is still flushed to the disk each second so you normally would not loose more than 1-2 sec worth of updates. Value 0 is a bit faster but is a bit less secure as you can lose transactions even in case MySQL Server crashes. Value 2 only cause data loss with full OS crash. If you are importing or altering cdr table it is strongly recommended to set temporarily innodb_flush_log_at_trx_commit = 0 and turn off binlog if you are importing CDR via inserts.
innodb_flush_log_at_trx_commit = 2
compression
MySQL 5.1
set in my.cf in [global] section this value:
innodb_file_per_table = 1
MySQL > 5.1
MySQL> set global innodb_file_per_table = 1; MySQL> set global innodb_file_format = barracuda;
Tune KEY_BLOCK_SIZE
If you choose KEY_BLOCK_SIZE=2 instead of 8 the compression will be twice better but with CPU penalty on read. We have tested differences between no compression, 8kb and 2kb block size compression on 700 000 CDR with this result (on single core system – we do not know how it behaves on multi core systems). Testing query is select with group by.
No compression – 1.6 seconds 8kb - 1.7 seconds 4kb - 8 seconds
Read performance
Read performance depends how big the database is and how fast disk operates and how much memory is allocated for innodb cache. Since sniffer version 7 all large tables uses partitioning by days which reduces needs to allocate very large cache to get good performance for the GUI. Partitioning works since MySQL 5.1 and is highly recommended. It also allows instantly removes old data by wiping partition instead of DELETE rows which can take hours on very large tables (millions of rows).
innodb_buffer_pool_size
This is very important variable to tune if you’re using Innodb tables. Innodb tables are much more sensitive to buffer size compared to MyISAM. MyISAM may work kind of OK with default key_buffer_size even with large data set but it will crawl with default innodb_buffer_pool_size. Also Innodb buffer pool caches both data and index pages so you do not need to leave space for OS cache so values up to 70-80% of memory often make sense for Innodb only installations.
We recommend to set this value to 50% of your available RAM. 2GB at least, 8GB is optimal. All depends how many CDR do you have per day.
innodb_buffer_pool_size = 8GB
Partitioning
Partitioning is enabled by default since version 7. If you want to take benefit of it you need to start with clean database - there is no conversion procedure from old database to partitioned one. Just create new database and start voipmonitor with new database and partitioning will be created. You can turn off partitioning by setting cdr_partition = no in voipmonitor.conf