Trisul probes are running on hundreds of different networks and are generally very stable. We also put them though fuzz tests and mangled packets to test the stability. However, it is possible that sometimes the Probes stop. This document describes how you can debug such events.

If you find Trisul Probe has crashed, you can collect the following information before restarting and see if you find some clues.

1. Trisul probe log files
2. Version number of Trisul Probe
3. Platform (Ubuntu/CentOS version)
4. dmesg output
5. Some information about the frequency of crashes

The log files on the trisul probe are stored under /usr/local/var/log/trisul-probe/domain0/probe0/context0 The latest ns-xxx.log contains the most recent logs before the crash. You can also use the helper bash aliases :

source /usr/local/share/trisul-probe/trisbashrc 
 
# to change to probe log dir
cd.l
 
# to tailf probe log file
tailf.ns 
 
# to open the CLI
tp

Sometimes useful clues can be found in the kernel message log. For example, when trisul is stopped by an OOM - Out of Memory killer. To view dmesg output

# to dump the kernel message log 
dmesg -T 
 
# to view trisul messages 
dmesg -T | grep trisul

To get probe version

trisul –version

To get platform version

uname -a

To get installed packages on probe # on ubuntu dpkg -l | grep trisul # on centos rpm -qa | grep trisul