articles:livevspcap
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
articles:livevspcap [2017/11/15 18:53] – [Issue 1 : The Clock] veera | articles:livevspcap [2017/11/15 23:27] (current) – [Why read in PCAP files] veera | ||
---|---|---|---|
Line 5: | Line 5: | ||
===== Why read in PCAP files ===== | ===== Why read in PCAP files ===== | ||
- | When you have a large PCAP file , say covering 6 hours of traffic. You have two choices, you can choose to replay the file to a NSM system using a tool like TCPReplay. But that would take 6 hours !! If you speed it up or play it at top speed, you lose crucial information about traffic metrics. | + | When you have a large PCAP file , say covering 6 hours of traffic. You have two choices, you can choose to replay the file to a NSM system using a tool like TCPReplay. But if you did that at normal rate it would take 6 hours !! If you speed it up or play it at top speed, you lose crucial information about traffic metrics. |
But if you played it back at natural rate, then that is an enormous waste of CPU and time. You could be processing packets at the natural rate of 1MB/s for hours when your powerful CPU can do 800Mbps. | But if you played it back at natural rate, then that is an enormous waste of CPU and time. You could be processing packets at the natural rate of 1MB/s for hours when your powerful CPU can do 800Mbps. | ||
Line 12: | Line 12: | ||
===== Issue 1 : The Clock ===== | ===== Issue 1 : The Clock ===== | ||
- | In Live traffic capture, the wall time is the clock. During low traffic periods your CPU and Memory usage goes down, but the rate of time is fixed. | + | All packet capture tools I know are clocked by the timestamps present in the packets themselves. |
+ | |||
+ | In a Live traffic capture, the wall time is the clock. During low traffic periods your CPU and Memory usage goes down, but the time moves at fixed rate. Tick, tock.. | ||
+ | |||
+ | When you read PCAPs, the time moves faster or slower depending on how fast your tool can inject the PCAP files. | ||
+ | |||
+ | A process-A that generates info-A can rip through the PCAP file at 100Mbps, another process-B generating info-B may only be able to process at 5Mbps. | ||
- | When you read PCAPs, most tools are clocked on the timestamp present in each packet inside the PCAP file, not on the wall clock. | ||
- If Event-B and Event-A that occurred at the same time in real world, arrive at the backend 40 minutes apart. Can they be stored and indexed correctly ? | - If Event-B and Event-A that occurred at the same time in real world, arrive at the backend 40 minutes apart. Can they be stored and indexed correctly ? | ||
- If Event-B generates some new enrichment data about Event-A and they arrive 40 minutes late at the backend. What happens to the enrichment? | - If Event-B generates some new enrichment data about Event-A and they arrive 40 minutes late at the backend. What happens to the enrichment? | ||
+ | |||
+ | A lot of this depends on the capabilities of the database. | ||
+ | |||
+ | |||
+ | ===== Issue 2 : State ===== | ||
+ | |||
+ | Some protocols need to maintain state. A good example is FTP which has a control channel which sets up a data channel. Another is SSL Session resumption, or SIP/TRP for VOIP, etc. | ||
+ | |||
+ | Your tools needs to maintain this state in order to decode the protocols correctly. A single threaded processing pipeline may not need a state synchronization but a multi threaded pipe does. | ||
+ | |||
+ | Even for PCAP file reads, we can benefit by multiple threads. Trisul uses 2 threads by default, you can also use 4, or 8 threads to speed up PCAP file reads. The problem here is the following. | ||
+ | |||
+ | * If you have TWO threads of processing , now thread A and thread B need to synchronize their state every X packets or every X seconds, etc. | ||
+ | * Say you synchronize every 10ms. In a real world FTP capture, there will be enough lag between the control and data channels, so you can get away with this sync rate. | ||
+ | * In a PCAP import, you can have 1 control packet and immediately followed by a data packet that might go to another thread. Since you are reading as fast as you can, the state doesnt sync. | ||
+ | |||
+ | |||
+ | There are some other less important differences, | ||
articles/livevspcap.1510752233.txt.gz · Last modified: 2017/11/15 18:53 by veera