====== Debugging crashes and other problems on the Trisul Probe nodes ======

Despite our best efforts with testing , sometimes the Trisul-Probes can stop for a number of reasons. This document describes how you can debug such events. 

===== Where to look =====

If you find Trisul Probe has crashed, you can collect the following information before restarting.  If nothing jumps out , you can contact [[https://www.trisul.org/support/|Trisul Support]] and send us the logs.

==== Log files ====

The log files on the trisul probe are stored under ''/usr/local/var/log/trisul-probe/domain0/probe0/context0''  The latest ''ns-xxx.log'' contains the most recent logs before the crash. You can also use the [[https://www.trisul.org/docs/ref/trisbashrc.html|trisbashrc bash aliases]] to navigate the directories as shown below:  

<code bash>
# add the helper aliases
source /usr/local/share/trisul-probe/trisbashrc 

# to change to probe log dir
cd.l

# to tailf probe log file
tailf.ns 

# the latest log file
cd.l
ls -lrt ns*

</code>

When contacting Trisul Support send us the latest log file ns-xxx.log

==== dmesg : view messages from the kernel ====

Sometimes useful clues can be found in the kernel message log.  For example, when trisul is stopped by an OOM - Out of Memory killer.  To view dmesg output

<code bash>

# to dump the kernel message log 
dmesg -T 

# to view trisul messages 
dmesg -T | grep trisul

</code>

==== Software versions ====

To get probe version  ''trisul --version''

To get platform version ''uname -a'' 

To get installed packages on probe

<code bash>
# on ubuntu
dpkg -l | grep trisul

# on centos
rpm -qa | grep trisul

</code>


===== Debugging with a binary with symbols =====

If we find nothing in the Trisul and Kernel log files, we (Trisul Support) will send you a special binary image with symbols. You can then run it under ''gdb'' to help pinpoint the location of the crash. This could be a bug

<note tip>We recommend you run this under the 'screen' utility, so you can detach the terminal and log off. 
</note>


<code>

# install gdb
apt install gdb

# run the special trisul binary we send you like this 
gdb --args /home/mike3/trisul -nodemon \
    /usr/local/etc/trisul-probe/domain0/probe0/context0/trisulProbeConfig.xml \
       -mode online_rxring

(...wait...) 

# after it crashes - type bt on the gdb prompt

gdb:  bt

..output here.. send us this output

</code>