Overview

System Architecture

FMADIO VOIP

FMADIO VOIP Analytics decodes and splits VOIP traffic in realtime on all FMADIO Packet Capture hardware systems. It consists of small plugin with a full GUI for analysis.

10g packet capture architecture

Features:


  • Realtime VOIP Decoding
  • Realtime SIP / RTP Decoding
  • One Call, One PCAP download
  • Optional SIP Only PCAP Download
  • Fast Easy to use GUI
  • Expandable to most VOIP Protocols

Install

Installation

FMADIO VOIP Analytics is installed using our modular plugin architecture. Installation example as follows [email protected]:~$ plugin_reload.lua fmadio_voip_g711_20170706_2324.tcz fmad fmadlua Jul 7 2017 calibrating... 0 : 3500000238 3.5000 cycles/nsec offset:0.000 Mhz Cycles/Sec 3500000238.0000 Std: 0 cycle std( 0.00000000) Target:3.50 Ghz argv /opt/fmadio/bin/fmadiolua loading filename [/opt/fmadio/bin/plugin_reload.lua] Sat Jul 8 21:28:28 2017 Plugin Load Loading Plugin [fmadio_voip_g711_20170706_2324.tcz] MD5: f4d5121be5914406ea931f6e2235cf3c fmadio_voip_g711_20170706_2324.tcz reloading voip [g711] Copying new firmware [fmadio_voip_g711_20170706_2324.tcz] -> /mnt/sda1/tce/optional/fmadio_voip_g711.tcz Cmd [sudo cp -v fmadio_voip_g711_20170706_2324.tcz /mnt/sda1/tce/optional/fmadio_voip_g711.tcz] Cmd [cp /mnt/sda1/tce/onboot.lst /mnt/sda1/tce/onboot.lst.bak] Cmd [cat /mnt/sda1/tce/onboot.lst.bak | grep -v analytics > /mnt/sda1/tce/onboot.lst] Killing programs Cmd [sudo /usr/local/bin/umount /tmp/tcloop/fmadio_voip_g711] umount: /tmp/tcloop/fmadio_voip_g711: not mounted Cmd [sudo mkdir -p /tmp/tcloop/fmadio_voip_g711] Cmd [sudo /usr/local/bin/mount /mnt/sda1/tce/optional/fmadio_voip_g711.tcz /tmp/tcloop/fmadio_voip_g711 -t squashfs -o loop,ro,bs=4096] Cmd [yes | sudo cp -ais /tmp/tcloop/fmadio_voip_g711/* / 2>/dev/null ] ----------------------------------------------- Updated: -> market g711 -> 354 -> Thu Jul 6 23:24:44 2017 ----------------------------------------------- done 0.057625Sec 0.000960Min

The first time the plugin is loaded a full reboot is required. Other times the plugin reload might be required multiple times. This is due to not all processes and mounts completely unloading the first time (e.g. binaries still being used)

Configuration

Historical VOIP Analysis

To run historical VOIP analysis on previously capture data, the following command is used: /opt/fmadio/analytics/voip_g711_index.lua
Note that:

  • All previous VOIP call history will be erased
  • This process every capture on the system, unless --filter is specified
  • --filter is a wildcard match / grep match on the capture name. It can match multiple capture names.


Using the --filter option to selectively re-index a capture or range of captures. Start by listing the current captures on the system

[email protected]:/opt/fmadio/analytics$ sudo stream_dump Streams: [0000] [this should be empty] 0GB Chunk(Cnt: 0 Start: 1 End: 0) Inv:-nan Cap:-nan CacheI:-nan Cache:-nan Disk:-nan Drop:-nan Pkt:0 [0001] remote_download_1499426232761_20170707_2017 149GB Chunk(Cnt: 610510 Start: 29 End: 610538) Inv:0.000 Cap:0.000 CacheI:0.000 Cache:1.000 Disk:0.000 Drop:0.000 Pkt:2000000000 [0002] remote_download_1499426370829_20170707_2020 149GB Chunk(Cnt: 610510 Start: 610566 End: 1221075) Inv:0.000 Cap:0.000 CacheI:0.000 Cache:1.000 Disk:0.000 Drop:0.000 Pkt:2000000000 [0003] remote_download_1499426876664_20170707_2028 149GB Chunk(Cnt: 610510 Start: 1221103 End: 1831612) Inv:0.000 Cap:0.000 CacheI:0.000 Cache:1.000 Disk:0.000 Drop:0.000 Pkt:2000000000 [0004] remote_download_1499427389634_20170707_2036 149GB Chunk(Cnt: 610510 Start: 1831640 End: 2442149) Inv:0.000 Cap:0.000 CacheI:0.000 Cache:1.000 Disk:0.000 Drop:0.000 Pkt:2000000000 [0005] remote_download_1499427898355_20170707_2045 149GB Chunk(Cnt: 610510 Start: 2442177 End: 3052686) Inv:0.000 Cap:0.000 CacheI:0.000 Cache:1.000 Disk:0.000 Drop:0.000 Pkt:2000000000 [0006] remote_prefilter__1499429159692_20170707_2106 2GB Chunk(Cnt: 9572 Start: 3113817 End: 3123388) Inv:0.000 Cap:0.000 CacheI:0.000 Cache:1.000 Disk:0.000 Drop:0.000 Pkt:10850610 [email protected]:/opt/fmadio/analytics$

In this specific case we want to re-index only the capture named remote_download_1499427898355_20170707_2045
To re-index only the capture remote_download_1499427898355_20170707_2045 run the following [email protected]:/opt/fmadio/analytics$ sudo ./voip_g711_index.lua --filter remote_download_1499427898355_20170707_2045 fmad fmadlua Jul 7 2017 calibrating... 0 : 3499997611 3.5000 cycles/nsec offset:0.002 Mhz Cycles/Sec 3499997611.0000 Std: 0 cycle std( 0.00000000) Target:3.50 Ghz argv /opt/fmadio/bin/fmadiolua argv remote_download_1499427898355_20170707_2045 loading filename [./voip_g711_index.lua] G711 decoder Static Indexing StartTime: 20170708_215908 OpenCtrl [/opt/fmadio/status/analytics] (fSysAnalytics_t*) Length 1048576B Cmd[sudo killall g711_decoder] killall: g711_decoder: no process killed Cmd[sudo killall www_fcgivoip_g711] Cmd[sudo killall stream_cat] killall: stream_cat: no process killed Cmd[sudo rm -Rf /mnt/store0/protocol/g711/] Cmd[mkdir /mnt/store0/protocol/] Cmd[mkdir /mnt/store0/protocol/g711/] Adding Stream 5 @ [remote_download_1499427898355_20170707_2045] Cmd[sudo /opt/fmadio/bin/stream_cat remote_download_1499427898355_20170707_2045 | sudo /opt/fmadio/bin/g711_decoder --stdin --cpu 10 --stream_index 5 >> /mnt/store0/log/g711_index_20170708_215908 & ] reading PCAP from stdin StreamIndex: 5 calibrating... filename calibrating... 0 : 00000000d09dafe8 3.5000 cycles/nsec Cycles/Sec 3499995112.0000 Std: 0cycle std( 0.00000000) 0 : 3500819200 3.5008 cycles/nsec offset:0.819 Mhz Cycles/Sec 3500819200.0000 Std: 0 cycle std( 0.00000000) Target:3.50 Ghz Nano PACP [Sat Jul 8 21:59:09 2017] G711 stream_cat: true 31.10GB g711 true 0.08GB . . .
Which starts the re-indexing and may take 1 minute to several hours depending on the amount of data that needs to be processed. The GUI will be shutdown immediately after the command is issued and re-started in ~ 1 minute.

Running Analysis 24/7

Schedule 24/7 Operation

The VOIP Analytics are designed to run in "always on" 24/7 mode. It can also operate in a restricted hours / days mode if required. To setup navigate to the Config Menu as shown below in GREEN.

10g VOIP analytics config menu

Scroll down the Analytics Schedule as shown below

10g VOIP analytics config schdule

Add a new row and change the Analytics Engine from "newrow0" -> voip_g711_realtime Its critical to use this extract name, otherwise the analytics will not be started. Then setup to run 24/7 Monday - Sunday as shown below.

10g VOIP analytics config schdule voip realtime

At this point the voip analysis engine will spawn within ~ 1 minute (runs from a cronjob) and will begin processing the currently active capture, or wait until a capture has started.

VOIP System Status

Check Script

There are a number of way to monitor the health of the VOIP G.711 analysis software. The first check is ruining the built-in health check utility /opt/fmadio/analytics/voip_g711_status.lua The output is similar to this. You can see the GOOD status meaning the system is running as expected [email protected]:/opt/fmadio/analytics$ ./voip_g711_status.lua fmad fmadlua Jul 8 2017 calibrating... 0 : 3499993012 3.5000 cycles/nsec offset:0.007 Mhz Cycles/Sec 3499993012.0000 Std: 0 cycle std( 0.00000000) Target:3.50 Ghz argv /opt/fmadio/bin/fmadiolua loading filename [./voip_g711_status.lua] ---------------------------------------------------- Checking process`s are up Found process [voip_g711_realtime.lua] Found process [g711_decoder] Found process [www_fcgivoip_g711] Found process [g711_monitor] ---------------------------------------------------- Last Call ID: [email protected] Last Call Date: 20170705_224041 Last Call Start: 13:40:41.248.989.440 Last Call Stop: 13:40:41.702.597.707 Last Call Capture: remote_midroll_1499481861709971968_20170708_2359 ---------------------------------------------------- voip_analytics_status: GOOD ---------------------------------------------------- done 0.140214Sec 0.002337Min [email protected]:/opt/fmadio/analytics$

G.711 Logfiles

For a more detailed investigation log files are best. The primary log file is in /mnt/store0/log/g711_decoder.cur Running tail -F /mnt/store0/log/g711_decoder.cur in an SSH window provides details on each VOIP call which has been decoded. If nothing is being output to this file it means no calls are currently being decoded

An example is shown below
[email protected]:~$ tail -F /mnt/store0/log/g711_decoder.cur 13:01:34.080.041.984 22134 SIPSession CallID[[email protected]:5000] To[[email protected]] From[[email protected]] Src[x.x.x.x. [email protected]] Dst[ 0. 0. 0. [email protected]] Proto:RTP/AVP Format:8 13:01:34.525.178.880 22651 SIPSession CallID[[email protected]:5000] To[[email protected]] From[[email protected] ] Src[[email protected]] Dst[ 0. 0. 0. [email protected]] Proto:RTP/AVP Format:8 13:01:34.535.633.920 22652 SIPSession CallID[[email protected]:5000] To[[email protected]] From[[email protected]] Src[[email protected]] Dst[ 0. 0. 0. [email protected]] Proto:RTP/AVP Format:8 13:01:35.288.875.264 22653 SIPSession CallID[[email protected]:5000] To[[email protected]] From[[email protected]] Src[[email protected]] Dst[ 0. 0. 0. [email protected]] Proto:RTP/AVP Format:8

VOIP System Restart

Restarting

In the event of a significant problem or decode hang status it many be required to forcibly stop analytics processing and wait for the system to restart. This will process stops all relevant processes. /opt/fmadio/analytics/voip_g711_stop.lua The output is similar to this. [email protected]:/opt/fmadio/analytics$ ./voip_g711_stop.lua fmad fmadlua Jul 12 2017 calibrating... 0 : 3499995004 3.5000 cycles/nsec offset:0.005 Mhz Cycles/Sec 3499995004.0000 Std: 0 cycle std( 0.00000000) Target:3.50 Ghz argv /opt/fmadio/bin/fmadiolua loading filename [./voip_g711_stop.lua] killall: stream_cat: no process killed killall: g711_decoder: no process killed killall: g711_monitor: no process killed killall: voip_g711_realtime.lua: no process killed done 0.032480Sec 0.000541Min [email protected]:/opt/fmadio/analytics$

After running the above command the GUI and backend systems have stopped. This can be confirmed using the status utility /opt/fmadio/analytics/voip_g711_status.luaIf analytics is running using the scheduler it will re-spawn within ~ 1 minute.

Analysis

Analysis Dashboard

Dashboard

VOIP Analysis Dashboard provides high level statistics on the currently indexed calls. An example is shown below.

10g packet voip dashboard

In this example it shows


  • SIP Sessions Total : Total number of calls on indexed on the device
  • SIP Sessions Open : Total number of open calls indexed on the device
  • SIP Sessions Timeout : Total number calls that have timed out
  • Call First : Date and time of the first call currently indexed on the device
  • Call Last : Date and time of the last call currently indexed on the device
  • Call Period : Number of hours and minutes currently indexed on the box. Expected to be 100`s of hours


Disk Full

To clarify the disk storage model, FMAD Packet Capture systems operate in a FIFO disk storage mode. The disk never gets full, it rotates data off the device in a First In First Out mode. As a result capture data organically rolls off the device. The "Call First" statistic shows the last call that is currently on the device, its accuracy is +/- 5 minutes or so. And the Call Period shows the amount of wallclock time is on the device.

VOIP Calls

Fetching Calls

FMADIO VOIP Analytics Web GUI automatically fetches the latest 1000 calls on the system, the result is similar to the screenshot below.

10g packet capture architecture

To drill down in a bit more detail start with the "SIP Fetch" options. This is what actually gets fetched across from the FMAD VOIP Analytics Packet Capture system to your local machine. Where as the "SIP Sessions" filters are all performed in memory on your local machine with no data fetches from the FMADIO Device.

10g packet analysis sip fetch

The above image shows the SIP Fetch parameters. The search fields are all options. If a Time is specified without a Date the current date will be used.

At the end of the SIP Fetch bar are the follow icons shown below
10g packet analysis sip fetch icons

The GREEN arrow icon, this issues a single one shot fetch based on the current parameters.

The GREY double arrow icon, this turns on continuous update. It will re-fetch the data every 10 seconds automatically. When activated the icon becomes green.

The CSV Icon is to download the current SIP Session log in CSV format


VOIP Calls PCAP Download

Calls can be downloaded in PCAP format by clicking on the PCAP icons. The icon with SIP is for SIP only data (excluding the actual RTP / voice data). This can be very helpful to debug issues without having to download a large PCAP.

10g packet analysis sip fetch icons


The State field indicates the call state as follows

  • C - Call is Closed
  • O - Call is Open
  • T - Call has timed out

Other

FMADIO VOIP Analysis is a cost effective way to monitor and analyize G.711 based VOIP network packet traffic


1-8-3 shibuya toc 8th floor,tokyo,japan
Contact Us