Differences

This shows you the differences between two versions of the page.

--- scripting:introbro [2018/09/28 18:46] – [Two scripting pipelines in Trisul] veera
+++ scripting:introbro [2018/09/28 23:18] (current) – veera
@@ Line 1: / Line 1: @@
 ====== Introduction to Trisul Scripting for Bro IDS users ======
-Bro IDS is a popular open source network analysis platform. A key feature of Bro IDS is the custom BRO language that allows you to write scripts to enhance the functionality of the platform.  Trisul Network Analytics is also a platform that can be extended by writing scripts. This page introduces the Trisul Scripting API for those who are already familiar with Bro IDS scripting.
+A key feature of the popular open source network analysis platform Bro IDS is the custom BRO language. It allows you to write scripts to enhance the functionality of the platform.  Trisul Network Analytics is also a platform that can be extended by writing scripts. This page introduces the Trisul Scripting API for those who are already familiar with Bro IDS scripting.
-===== Trisul API =====
+===== Trisul scripting  =====
-==== Outputs : database objects vs logs ====
+There are two differences between the Bro and Trisul approaches. Lets sort them out before diving deeper.
+First, Trisul integrates the packet processing as well as the database.So you directly work with database objects like metrics , resources, flows, documents, and graphs.This can be a bit confusing to Bro scripters who focus on generating logs and notices.
-Trisul is built from ground up to be full streaming analytics platform - database included. In Trisul, you work directly with database objects like metrics , resources, flows, documents, and graphs.This can be a bit confusing to Bro scripters who focus on generating logs.
+A second difference is,  You can script either the packet processing stream or the analytics stream. We call these two streams the [[https://www.trisul.org/docs/lua/basics.html#frontend_and_backend_scripts|Frontend (Fastpath)  or the Backend (slowpath)]].  They communicate using a messaging API. This can be a bit odd to newcomers because the backend scripts have a longer time budget to execute.
-To illustrate with an example.
-**Say you are calculating TLS Fingerprints from network traffic**
-  * In Bro, you might write scripts to add the fingerprint to the connection/flow log.
-  * In Trisul, your approach would be to create a new counter group for TLS Fingerprints and count each print there. You can also mark the flows like Bro, or create graph edges, but the main focus is on metrics.
-==== Two scripting pipelines in Trisul  ====
-A second architectural difference is : In Trisul, you can script either the packet processing stream or the analytics stream. We call these two streams the Frontend (Fastpath)  or the Backend (slowpath).  The Frontend / Fastpath scripts work on packets and reassembled payloads, and the Backend scripts work on objects like traffic metrics for a particular entity, Top-K,  flows, resources, etc.   The two pipelines can talk to each other using a messaging API.
 === Comparison ===
-|Feature | Bro | Trisul |
+^ Feature ^ Bro ^ Trisul ^
-|language | .bro language | lua |
+|language | .bro language | LuaJIT  |
-|protocol decoding | Bro framework provides fine grained events representing protocol fields to your script.  | Trisul framework provides a lower level access to the raw payload bytes for a protocol. You have to decode it yourself. It is not as hard as it sounds, you can use the BITMAUL library to dissect protocols to the depth you want |
+|docs|[[https://www.bro.org/sphinx/scripting/index.html#understanding-bro-scripts|Bro Scripting]]|[[https://www.trisul.org/docs/lua/|Trisul LUA API]] |
-|events | fine grained "typed" events. For example ''dns_A6_reply(..)'' event contains parsed fields for the DNS AAAA reply record |  loose documents in a canonical text format.  In Trisul, //DNS Resource// is a text dump of a DNS transaction in a canonical DIG format. You can pick the fields you want using Regex. This means you have a dramatically lower number of events to deal with and are free to decode packets to the depth you want.   |
+|protocol decoding | Bro framework provides fine grained events representing protocol fields to your script.  | Trisul framework provides a lower level access to the payload itself, or for some common protocols the results of Trisul's built in dissection. Decoding a payload isnt as hard as it sounds, we released the open source [[https://github.com/trisulnsm/bitmaul|BITMAUL library]] to dissect protocols to the depth you want. |
+|events | fine grained "typed" events. For example ''dns_A6_reply(..)'' event contains parsed fields for the DNS AAAA reply record |loose documents in a canonical text format.  In Trisul, //DNS Resource// is a text dump of a DNS transaction in a canonical DIG format. You can pick the fields you want using Regex. This means you have a dramatically lower number of events to deal with and are free to decode packets to the depth you want.   |
+|extending | you can write C code and integrate it to your Bro scripting using a *.bif file. This involves a binary compile process |leverages LuaJIT FFI to directly call library functions |
+|time budget | packet drops if script is slow |frontend scripts have to execute fast to avoid packet drops, backend scripts have  a more relaxed time budget of about 0-30 seconds. |
+|deployment|place script files in directory| similar place scripts in directory on probe or put it on the hub node for automatic deployment to all probes |
+|debugger| manual | built in LUA debugger. just call ''dbg()'' to drop into an interactive debugger|
+|threading|single with load balanced workers | multithreaded with load balanced threads, this allows for very fast state sharing between multiple threads using message passing. During development and debugging you can turn it into a single threaded system |
+|async|yes|yes - you can have a deferred execution of a LUA code block|
+|intel|intel framework|you can choose your own framework. We like to dump all threat intel into a LevelDB database using LuaJIT FFI to access LevelDB. You can choose any other system. |
+|packaging|Yes - Bro packages| Yes - Trisul APPs |
+|example|JA3 TLS Fingerprint written [[ https://github.com/salesforce/ja3/tree/master/bro|in Bro]]  | JA3 [[https://github.com/trisulnsm/apps/blob/master/analyzers/tls-print/jahash.lua|written in Trisul]] notice how in Trisul we parse the TLS record manually, while in BRO we use the typed events like  ''ssl_client_hello()'' , ''ssl_extensions()'' etc which are supplied by Bro. With Trisul, you have slightly more work to do with the parsing the protocol, but you are independent of what the framework supplies. The Trisul code is longer because we are adding a lot of metrics and graph analytics in the script |
+|disadvantage| - | LuaJIT has a 2GB limit on total memory use, therefore your scripts cant allocate too much memory. Use Trisul aggregations instead of building large lookup tables or use LevelDB to store data. |
+===== Types of Trisul Scripts =====
+There are 16 different types of Trisul scripts depending on what you want to accomplish.
+  * **6 Frontend scripts** that work on Packets and Reassembly : Input Filter , Simple Counter, Reassembly Handler, File Extraction , PCAP storage , Protocol Handler,
+  * **7 Backend scripts** that work on metrics, documents, resources : Engine Monitor, Alert Monitor, Counter Monitor, Session Monitor, Resource Monitor, FTS Monitor , Flow Tracker
+  * **3 Definition scripts** : New Counter Group, New Alert Group, New Resource Group
+For more on this refer to [[https://www.trisul.org/docs/lua/selector.html|Script selector cheat sheet]]
+To conclude, this was a quick introduction to Trisul scripting. We will be diving deeper into each of the areas in the coming days.