Results 1 to 20 of 20
  1. #1
    Join Date
    Jul 2007
    Location
    Virginia
    Posts
    1,314

    Flow analysis tools

    I did a quick search but it looks like most of the threads are rather old.

    Any thoughts on flow analyzers? I've been looking at sFlow / Netflow analyzers like Scrutinizer lately and would like to some feedback on it as well as insights into other stuff that's on the market.
    ~ @PreetamJinka

  2. #2
    Join Date
    May 2005
    Location
    Bay Area
    Posts
    1,211
    Try NFSEN. If you're reasonably technical, you can get that setup and have a great, free flow analysis platform.

    Morgan

  3. #3
    Join Date
    Jul 2013
    Location
    Munich - Europe - Germany
    Posts
    11
    Quote Originally Posted by Preetam View Post
    I did a quick search but it looks like most of the threads are rather old.

    Any thoughts on flow analyzers? I've been looking at sFlow / Netflow analyzers like Scrutinizer lately and would like to some feedback on it as well as insights into other stuff that's on the market.
    Write the data into SQL database and use normal SQL queries after it...

  4. #4
    Join Date
    Jul 2007
    Location
    Virginia
    Posts
    1,314
    Quote Originally Posted by Datacentre-MUC View Post
    Write the data into SQL database and use normal SQL queries after it...
    I'm looking for tools that will let me avoid this.

  5. #5
    Join Date
    Jun 2001
    Location
    Denver, CO
    Posts
    3,302
    Quote Originally Posted by Preetam View Post
    I did a quick search but it looks like most of the threads are rather old.

    Any thoughts on flow analyzers? I've been looking at sFlow / Netflow analyzers like Scrutinizer lately and would like to some feedback on it as well as insights into other stuff that's on the market.
    What are your major objectives? We actually take highly sampled Juniper firewall filter log data into splunk for a realtime view into network activity at an aggregate level, primarily used for detecting anomalous data flows (DDoS and the like). We layer that with noction's reporting for more of the BGP type data (Traffic to various ASNs, etc ...).

    We've used Scrutinizer in the past but it's costly and the free version don't provide much in the way of historical data. The nice thing about Splunk is we have less than 500MB of logs per day, so we can use the free version and it's very easy to query any given IP to see historical traffic trends.
    Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
    AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
    Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
    Current specials here. Check them out.

  6. #6
    Join Date
    Jul 2007
    Location
    Virginia
    Posts
    1,314
    Quote Originally Posted by Jay Suds View Post
    What are your major objectives?
    One would be to detect anomalous flows through, for example, TCP packet header inspection. Looking for things like SYN floods, spoofed packets, etc.

    I don't have other specific requirements or objectives. I'm just trying to get a feel for the tools out there.

    Thanks for your input!

  7. #7
    Join Date
    Jul 2009
    Location
    The backplane
    Posts
    1,788
    nfcapd/nfdump/nfsen are fantastic for flow collection and analysis.

  8. #8
    Join Date
    Jan 2004
    Location
    Pennsylvania
    Posts
    942
    +1 for nfsen. It's not the prettiest, but it works well for exactly what you want -- finding the cause of traffic bursts.
    Matt Ayres - togglebox.com
    Linux and Windows Cloud Virtual Datacenters powered by Onapp / Xen
    Instant Setup, Instant Scalability, Full Lifecycle Hosting Solutions

    www.togglebox.com

  9. #9
    Join Date
    Sep 2012
    Posts
    253
    flow-tools and FlowViewer for us. We've tried splunk/scrutinizer/nfsen/ntop and in the end those two worked best for us. FlowViewer let's us run reports as needed. Then flow-tools give us the programmatic tools we need to setup alerts to our noc for potential problems without having to manually be running reports.

  10. #10
    Join Date
    Sep 2008
    Location
    Melbourne Australia
    Posts
    81
    Winding back the clock say 4 or years ago I remember trying lots of flow analysis software and evaluating lots of options with one goal in mind…

    Find attack traffic and quickly identify the source and destination along with the protocol in near real time, enabling us to lower the time it took to deal with threats, relying on SNMP data for this purpose was useless.

    In the end we choose ManageEngine Netflow Analyzer which provided a fantastic starting point for us in providing real time visibility, whilst now a days we NSFOCUS hardware mainly for Flow and DDoS detection and mitigation we still to this day use ManageEngine Netflow Analyzer within our NOC !

    A very old case study of mine can be found here - http://micron21.com/ddos-netflow.php

    Whilst the software is commercial I believe it’s still very well priced, and the free version from memory supports a single interface for free !

    Back in the days the software developers were very helpful in helping create custom modifications building new features and functions for us so well worth in my eyes checking it out !

    Kindest Regards

    James
    Melbourne Australia Datacentre – Micron21 Pty Ltd
    Co Location – Dedicated Servers – Web Hosting – Custom Solutions – 24/7 Support - AS 38880
    Phone 1300 76 99 72 Email - James @ micron21.com
    www.micron21.com

  11. #11
    You could just write your own The code to detect your average reflection attack isn't very hard!

    Our first iteration was a hacky shell script that used pmacct to handle all the traffic counting and netflow parsing. This actually worked pretty well, though we had some problems scaling it.

    We moved onto some custom code to parse everything and trigger mitigation. The basics of this are really easy to get into.

  12. #12
    Join Date
    Jul 2008
    Location
    Dallas, TX
    Posts
    107
    Another +1 for nfsen. We are using it as a framework and have written APIs to export the flow data to be presented in other interfaces besides the default nfsen webpage.
    Ryan G. - Limestone Networks - Network Engineer
    Cloud, Dedicated, & Enterprise Hosting - Premium Network - Passionate Support
    Resell Dedicated Servers - @LimestoneInc - 877.586.0555 x1

  13. #13
    Join Date
    Aug 2004
    Location
    Kauai, Hawaii
    Posts
    3,799
    I just spent the weekend working with flow tools and nfdump. Nfdump is the best solution and takes two minutes to install, the cli option on it's nfcapd tool are good and allows automated execution of your own script upon file rotation.

    Nfcapd listens for x time
    Script executes on rotation
    Presumably call nfdump (has nice csv output option) from your script
    Export data to SQL
    Repeat every x minutes

    -sent from ipad
    Last edited by gordonrp; 02-20-2014 at 10:36 AM. Reason: Typos on ipad

  14. #14
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Quote Originally Posted by gordonrp View Post
    I just spent the weekend working with flow tools and nfdump. Nfdump is the best solution and takes two minutes to install, the cli option on it's nfcapd tool are good and allows automated execution of your own script upon file rotation.

    Nfcapd listens for x time
    Script executes on rotation
    Presumably call nfdump (has nice csv output option) from your script
    Export data to SQL
    Repeat every x minutes

    -sent from ipad
    Nice summary. I rolled my own a couple years ago completely from scratch but it only works with sflow and only brocade devices at that -- haven't been able to figure out why it can't parse sflows from other vendors. Nfcapd seems like a good starting point for something better that will work with sflow and netflow.
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  15. #15
    Join Date
    Nov 2012
    Posts
    428
    Scrutinizer is a nice product but it gets really pricey once you start adding nodes.

  16. #16
    Join Date
    Aug 2004
    Location
    Kauai, Hawaii
    Posts
    3,799
    Quote Originally Posted by FastServ View Post
    Nice summary. I rolled my own a couple years ago completely from scratch but it only works with sflow and only brocade devices at that -- haven't been able to figure out why it can't parse sflows from other vendors. Nfcapd seems like a good starting point for something better that will work with sflow and netflow.
    Yeah that's the reason I spent the weekend on it rather than an hour. It took me a while to realize that jflow v8/v9 does not work with flow-tools, but nfdump does support the later versions. If you switch to nfdump you may find your scripts work on other devices, simply as it supports more versions of netflow/jflow/sflow (specifically the template based outputs).

  17. #17
    Join Date
    Jul 2009
    Location
    The backplane
    Posts
    1,788
    Quote Originally Posted by FastServ View Post
    haven't been able to figure out why it can't parse sflows from other vendors.
    "Standards" tend to be loosely implemented in many flow accounting implementations.

  18. #18
    Join Date
    Jul 2007
    Location
    Virginia
    Posts
    1,314
    Quote Originally Posted by Microlinux View Post
    "Standards" tend to be loosely implemented in many flow accounting implementations.
    Really? sFlow is a pretty straightforward spec. I can't see why vendors/tools would deviate from it.

  19. #19
    Join Date
    Aug 2006
    Location
    Ashburn VA, San Diego CA
    Posts
    4,615
    Quote Originally Posted by Preetam View Post
    Really? sFlow is a pretty straightforward spec. I can't see why vendors/tools would deviate from it.
    The issue in my case is most likely in the libraries I've been using. Originally written many years ago by AMSIX for their brocade gear (and somewhat sloppily, at that) and never updated since. It's one of those coding styles nobody except the author would touch without starting over. Fortunately we only need to be parsing brocade sflow for the time being but this thread has motivated me to look further .
    Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
    Since 2003 - Ashburn VA + San Diego CA Datacenters

  20. #20
    Join Date
    Jul 2007
    Location
    Virginia
    Posts
    1,314
    Do people export flows from TOR switches or routers? Or both? I don't see why you'd need exports from both since that seems redundant.

Similar Threads

  1. Security log analysis tools
    By AndyB78 in forum Hosting Security and Technology
    Replies: 2
    Last Post: 08-04-2011, 10:04 AM
  2. Which are the best & free tools for doing competitor SEO analysis?
    By jiten702 in forum SEO / SEM Discussions
    Replies: 2
    Last Post: 07-02-2010, 03:50 PM
  3. Complete Site Analysis - Link Popularity and Domain Analysis
    By elsonar in forum Software & Scripts Offers
    Replies: 0
    Last Post: 02-15-2009, 02:20 PM
  4. One all encompassing Online Website Analysis Tools
    By jphayes in forum Programming Discussion
    Replies: 2
    Last Post: 07-27-2007, 12:22 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •