Differences

This shows you the differences between two versions of the page.

--- articles:livevspcap [2017/11/15 18:35] – created veera
+++ articles:livevspcap [2017/11/15 23:26] – [Issue 1 : The Clock] veera
@@ Line 12: / Line 12: @@
 ===== Issue 1 : The Clock =====
-In Live traffic capture, the wall time is the clock. During low traffic periods your CPU and Memory usage goes down, but the rate of time is fixed.
+All packet capture tools I know are clocked by the timestamps present in the packets themselves.
+In a Live traffic capture, the wall time is the clock. During low traffic periods your CPU and Memory usage goes down, but the time moves at fixed rate. Tick, tock..
+When you read PCAPs, the time moves faster or slower depending on how fast your tool can inject the PCAP files.
+A process-A that generates info-A can rip through the PCAP file at 100Mbps, another process-B generating info-B may only be able to process at 5Mbps.  If both of these information streams hit a single backend, then we may have these problems.
+  - If Event-B and Event-A that occurred at the same time in real world, arrive at the backend 40 minutes apart. Can they be stored and indexed correctly ?
+  - If Event-B generates some new enrichment data about Event-A and they arrive 40 minutes late at the backend. What happens to the enrichment?
+A lot of this depends on the capabilities of the database.
+===== Issue 2 : State =====
+Some protocols need to maintain state. A good example is FTP which has a control channel which sets up a data channel. Another is SSL Session resumption, or SIP/TRP for VOIP, etc.
+Your tools needs to maintain this state in order to decode the protocols correctly. A single threaded processing pipeline may not need a state synchronization but a multi threaded pipe does.
+Even for PCAP file reads, we can benefit by multiple threads. Trisul uses 2 threads by default, you can also use 4, or 8 threads to speed up PCAP file reads. The problem here is the following.
+  * If you have TWO threads of processing , now thread A and thread B need to synchronize their state every X packets or every X seconds, etc.
+  * Say you synchronize every 10ms.  In a real world FTP capture, there will be enough lag between the control and data channels, so you can get away with this sync rate.
+  * In a PCAP import, you can have 1 control packet and immediately followed by a data packet that might go to another thread. Since you are reading as fast as you can, the state doesnt sync.
+There are some other less important differences, we can document in the next article.
-When you read PCAPs, most tools are clocked on the timestamp present in the PCAP file, not on the wall clock.  So it comes down to this rule.
-  * A single