Differences

This shows you the differences between two versions of the page.

--- tips:flowindextuning [2023/09/27 14:35] – created veera
+++ tips:flowindextuning [2023/09/27 15:04] (current) – [Types and typical sizes at ISP scale] veera
@@ Line 4: / Line 4: @@
 The Trisul flow database is a high performance high throughput store specifically designed for Network Flow storage.
-The tuning parameters are controlled by the following section in [[trisulHubConfig.xml Trisul Hub Configuration file]|https://www.trisul.org/docs/ref/trisulhubconfig.html#advanced_db_parameters ]
+The tuning parameters are controlled by the following section in [[https://www.trisul.org/docs/ref/trisulhubconfig.html#advanced_db_parameters |trisulHubConfig.xml Trisul Hub Configuration file]]
+===== Flow database tuning parameters =====
+<code lang=xml>
+<DBParameters>
+    <FlowStream>
+        <MicroSecondTimestamps>true</MicroSecondTimestamps>
+        <kFLOWS_PER_BLOCK></kFLOWS_PER_BLOCK>
+        <kBLOOM_AGG_SIZE></kBLOOM_AGG_SIZE>
+        <kBUMPX_AGG_SIZE></kBUMPX_AGG_SIZE>
+        <ZFLOWBLOCK_COMPRESSOR_CODE></ZFLOWBLOCK_COMPRESSOR_CODE>
+        <!-- Bitmap indices - to disable set buckets to 0 -->
+        <kHOST_BUMPX_BUCKETS></kHOST_BUMPX_BUCKETS>
+        <kAPP_BUMPX_BUCKETS></kAPP_BUMPX_BUCKETS>
+        <kPROTO_BUMPX_BUCKETS></kPROTO_BUMPX_BUCKETS>
+        <kGEN1_BUMPX_BUCKETS></kGEN1_BUMPX_BUCKETS>
+        <kGEN2_BUMPX_BUCKETS></kGEN2_BUMPX_BUCKETS>
+        <kGEN3_BUMPX_BUCKETS></kGEN3_BUMPX_BUCKETS>
+        <kAUX0_BUMPX_BUCKETS></kAUX0_BUMPX_BUCKETS>
+        <kAUX1_BUMPX_BUCKETS></kAUX1_BUMPX_BUCKETS>
+        <kAUX2_BUMPX_BUCKETS></kAUX2_BUMPX_BUCKETS>
+        <kHOST24_BUMPX_BUCKETS></kHOST24_BUMPX_BUCKETS>
+        <kHOST16_BUMPX_BUCKETS></kHOST16_BUMPX_BUCKETS>
+        <!-- Bloom filter matrix.. set to bits and hashes to 0 to disable an index -->
+        <kHOST_BLOOM_BITS>  </kHOST_BLOOM_BITS>
+        <kHOST_BLOOM_HASHES> </kHOST_BLOOM_HASHES>
+        <kAPP_BLOOM_BITS> </kAPP_BLOOM_BITS>
+        <kAPP_BLOOM_HASHES></kAPP_BLOOM_HASHES>
+        <kPROTO_BLOOM_BITS></kPROTO_BLOOM_BITS>
+        <kPROTO_BLOOM_HASHES></kPROTO_BLOOM_HASHES>
+        <kGEN1_BLOOM_BITS></kGEN1_BLOOM_BITS>
+        <kGEN1_BLOOM_HASHES> </kGEN1_BLOOM_HASHES>
+        <kGEN2_BLOOM_BITS> </kGEN2_BLOOM_BITS>
+        <kGEN2_BLOOM_HASHES> </kGEN2_BLOOM_HASHES>
+        <kGEN3_BLOOM_BITS> </kGEN3_BLOOM_BITS>
+        <kGEN3_BLOOM_HASHES> </kGEN3_BLOOM_HASHES>
+        <kAUX0_BLOOM_BITS> </kAUX0_BLOOM_BITS>
+        <kAUX0_BLOOM_HASHES> </kAUX0_BLOOM_HASHES>
+        <kAUX1_BLOOM_BITS> </kAUX1_BLOOM_BITS>
+        <kAUX1_BLOOM_HASHES> </kAUX1_BLOOM_HASHES>
+        <kAUX2_BLOOM_BITS> </kAUX2_BLOOM_BITS>
+        <kAUX2_BLOOM_HASHES> </kAUX2_BLOOM_HASHES>
+        <kHOST16_BLOOM_BITS> </kHOST16_BLOOM_BITS>
+        <kHOST16_BLOOM_HASHES> </kHOST16_BLOOM_HASHES>
+        <kHOST24_BLOOM_BITS> </kHOST24_BLOOM_BITS>
+        <kHOST24_BLOOM_HASHES> </kHOST24_BLOOM_HASHES>
+    </FlowStream>
+</DBParameters>
+</code>
+===== Indices and Filters =====
+The main tuning are the sizes of the Bitmap Indices and Bloom Filters. The details of the implementation are not important here. We talk about the tyeps and how to disable them.
+<note important>If a Index or Filter for a particular type is disabled, the query performance with that type shall be slower. </note>
+==== Types and typical sizes at ISP scale ====
+=== Bitmap indices ===
+|kHOST_BUMPX_BUCKETS| ip address v4 and v6| 870M + 870M + 870M  = 2.6G | 2.6G K1A + 2.6G K1Z  + 2.6G K21= 7.8GG |
+|kAPP_BUMPX_BUCKETS| ports | 300MB  | 4G |
+|kPROTO_BUMPX_BUCKETS| protocol |
+|kGEN1_BUMPX_BUCKETS| router IP |
+|kGEN2_BUMPX_BUCKETS| router Interface ingres|
+|kGEN3_BUMPX_BUCKETS| router interface egres|
+|kAUX0_BUMPX_BUCKETS| tag 0 |
+|kAUX1_BUMPX_BUCKETS| tag 1 |
+|kAUX2_BUMPX_BUCKETS| the main flow tag strings |
+|kHOST24_BUMPX_BUCKETS| net/24 | 860M | 2.6G|
+|kHOST16_BUMPX_BUCKETS| net/16 | 430M  | 670M|