Debugging crashes and other problems on the Trisul Probe nodes

Despite our best efforts with testing , sometimes the Trisul-Probes can stop for a number of reasons. This document describes how you can debug such events.

Where to look

If you find Trisul Probe has crashed, you can collect the following information before restarting. If nothing jumps out , you can contact Trisul Support and send us the logs.

Log files

The log files on the trisul probe are stored under /usr/local/var/log/trisul-probe/domain0/probe0/context0 The latest ns-xxx.log contains the most recent logs before the crash. You can also use the trisbashrc bash aliases to navigate the directories as shown below:

# add the helper aliases
source /usr/local/share/trisul-probe/trisbashrc 
# to change to probe log dir
# to tailf probe log file
# the latest log file
ls -lrt ns*

When contacting Trisul Support send us the latest log file ns-xxx.log

dmesg : view messages from the kernel

Sometimes useful clues can be found in the kernel message log. For example, when trisul is stopped by an OOM - Out of Memory killer. To view dmesg output

# to dump the kernel message log 
dmesg -T 
# to view trisul messages 
dmesg -T | grep trisul

Software versions

To get probe version trisul –version

To get platform version uname -a

To get installed packages on probe

# on ubuntu
dpkg -l | grep trisul
# on centos
rpm -qa | grep trisul

Debugging with a binary with symbols

If we find nothing in the Trisul and Kernel log files, we (Trisul Support) will send you a special binary image with symbols. You can then run it under gdb to help pinpoint the location of the crash. This could be a bug

We recommend you run this under the 'screen' utility, so you can detach the terminal and log off.
# install gdb
apt install gdb

# run the special trisul binary we send you like this 
gdb --args /home/mike3/trisul -nodemon \
    /usr/local/etc/trisul-probe/domain0/probe0/context0/trisulProbeConfig.xml \
       -mode online_rxring


# after it crashes - type bt on the gdb prompt

gdb:  bt

..output here.. send us this output
admin/debuggingcrash.txt · Last modified: 2018/10/03 18:50 by veera