Results 1 to 20 of 20
Thread: Flow analysis tools
-
02-14-2014, 02:20 AM #1Web Hosting Master
- Join Date
- Jul 2007
- Location
- Virginia
- Posts
- 1,314
Flow analysis tools
I did a quick search but it looks like most of the threads are rather old.
Any thoughts on flow analyzers? I've been looking at sFlow / Netflow analyzers like Scrutinizer lately and would like to some feedback on it as well as insights into other stuff that's on the market.~ @PreetamJinka
-
02-15-2014, 07:45 PM #2Web Hosting Master
- Join Date
- May 2005
- Location
- Bay Area
- Posts
- 1,211
Try NFSEN. If you're reasonably technical, you can get that setup and have a great, free flow analysis platform.
Morgan
-
02-16-2014, 08:36 AM #3Temporarily Suspended
- Join Date
- Jul 2013
- Location
- Munich - Europe - Germany
- Posts
- 11
-
02-16-2014, 12:40 PM #4Web Hosting Master
- Join Date
- Jul 2007
- Location
- Virginia
- Posts
- 1,314
-
02-16-2014, 01:41 PM #5Web Hosting Master
- Join Date
- Jun 2001
- Location
- Denver, CO
- Posts
- 3,302
What are your major objectives? We actually take highly sampled Juniper firewall filter log data into splunk for a realtime view into network activity at an aggregate level, primarily used for detecting anomalous data flows (DDoS and the like). We layer that with noction's reporting for more of the BGP type data (Traffic to various ASNs, etc ...).
We've used Scrutinizer in the past but it's costly and the free version don't provide much in the way of historical data. The nice thing about Splunk is we have less than 500MB of logs per day, so we can use the free version and it's very easy to query any given IP to see historical traffic trends.Jay Sudowski // Handy Networks LLC // Co-Founder & CTO
AS30475 - Level(3), HE, Telia, XO and Cogent. Noction optimized network.
Offering Dedicated Server and Colocation Hosting from our SSAE 16 SOC 2, Type 2 Certified Data Center.
Current specials here. Check them out.
-
02-16-2014, 05:08 PM #6Web Hosting Master
- Join Date
- Jul 2007
- Location
- Virginia
- Posts
- 1,314
One would be to detect anomalous flows through, for example, TCP packet header inspection. Looking for things like SYN floods, spoofed packets, etc.
I don't have other specific requirements or objectives. I'm just trying to get a feel for the tools out there.
Thanks for your input!
-
02-17-2014, 02:02 PM #7Web Hosting Master
- Join Date
- Jul 2009
- Location
- The backplane
- Posts
- 1,788
nfcapd/nfdump/nfsen are fantastic for flow collection and analysis.
-
02-17-2014, 05:30 PM #8Web Hosting Master
- Join Date
- Jan 2004
- Location
- Pennsylvania
- Posts
- 942
+1 for nfsen. It's not the prettiest, but it works well for exactly what you want -- finding the cause of traffic bursts.
Matt Ayres - togglebox.com
Linux and Windows Cloud Virtual Datacenters powered by Onapp / Xen
Instant Setup, Instant Scalability, Full Lifecycle Hosting Solutions
www.togglebox.com
-
02-18-2014, 12:59 AM #9Web Hosting Guru
- Join Date
- Sep 2012
- Posts
- 253
flow-tools and FlowViewer for us. We've tried splunk/scrutinizer/nfsen/ntop and in the end those two worked best for us. FlowViewer let's us run reports as needed. Then flow-tools give us the programmatic tools we need to setup alerts to our noc for potential problems without having to manually be running reports.
-
02-18-2014, 06:45 PM #10Junior Guru Wannabe
- Join Date
- Sep 2008
- Location
- Melbourne Australia
- Posts
- 81
Winding back the clock say 4 or years ago I remember trying lots of flow analysis software and evaluating lots of options with one goal in mind…
Find attack traffic and quickly identify the source and destination along with the protocol in near real time, enabling us to lower the time it took to deal with threats, relying on SNMP data for this purpose was useless.
In the end we choose ManageEngine Netflow Analyzer which provided a fantastic starting point for us in providing real time visibility, whilst now a days we NSFOCUS hardware mainly for Flow and DDoS detection and mitigation we still to this day use ManageEngine Netflow Analyzer within our NOC !
A very old case study of mine can be found here - http://micron21.com/ddos-netflow.php
Whilst the software is commercial I believe it’s still very well priced, and the free version from memory supports a single interface for free !
Back in the days the software developers were very helpful in helping create custom modifications building new features and functions for us so well worth in my eyes checking it out !
Kindest Regards
JamesMelbourne Australia Datacentre – Micron21 Pty Ltd
Co Location – Dedicated Servers – Web Hosting – Custom Solutions – 24/7 Support - AS 38880
Phone 1300 76 99 72 Email - James @ micron21.com
www.micron21.com
-
02-18-2014, 10:23 PM #11Junior Guru
- Join Date
- Dec 2013
- Posts
- 194
You could just write your own The code to detect your average reflection attack isn't very hard!
Our first iteration was a hacky shell script that used pmacct to handle all the traffic counting and netflow parsing. This actually worked pretty well, though we had some problems scaling it.
We moved onto some custom code to parse everything and trigger mitigation. The basics of this are really easy to get into.
-
02-19-2014, 03:12 PM #12WHT Addict
- Join Date
- Jul 2008
- Location
- Dallas, TX
- Posts
- 107
Another +1 for nfsen. We are using it as a framework and have written APIs to export the flow data to be presented in other interfaces besides the default nfsen webpage.
Ryan G. - Limestone Networks - Network Engineer
Cloud, Dedicated, & Enterprise Hosting - Premium Network - Passionate Support
Resell Dedicated Servers - @LimestoneInc - 877.586.0555 x1
-
02-20-2014, 10:30 AM #13Corporate Member
- Join Date
- Aug 2004
- Location
- Kauai, Hawaii
- Posts
- 3,799
I just spent the weekend working with flow tools and nfdump. Nfdump is the best solution and takes two minutes to install, the cli option on it's nfcapd tool are good and allows automated execution of your own script upon file rotation.
Nfcapd listens for x time
Script executes on rotation
Presumably call nfdump (has nice csv output option) from your script
Export data to SQL
Repeat every x minutes
-sent from ipadLast edited by gordonrp; 02-20-2014 at 10:36 AM. Reason: Typos on ipad
-
02-20-2014, 02:38 PM #14Randy
- Join Date
- Aug 2006
- Location
- Ashburn VA, San Diego CA
- Posts
- 4,615
Nice summary. I rolled my own a couple years ago completely from scratch but it only works with sflow and only brocade devices at that -- haven't been able to figure out why it can't parse sflows from other vendors. Nfcapd seems like a good starting point for something better that will work with sflow and netflow.
Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
Since 2003 - Ashburn VA + San Diego CA Datacenters
-
02-20-2014, 02:43 PM #15Aspiring Evangelist
- Join Date
- Nov 2012
- Posts
- 428
Scrutinizer is a nice product but it gets really pricey once you start adding nodes.
-
02-20-2014, 02:46 PM #16Corporate Member
- Join Date
- Aug 2004
- Location
- Kauai, Hawaii
- Posts
- 3,799
Yeah that's the reason I spent the weekend on it rather than an hour. It took me a while to realize that jflow v8/v9 does not work with flow-tools, but nfdump does support the later versions. If you switch to nfdump you may find your scripts work on other devices, simply as it supports more versions of netflow/jflow/sflow (specifically the template based outputs).
-
02-20-2014, 06:20 PM #17Web Hosting Master
- Join Date
- Jul 2009
- Location
- The backplane
- Posts
- 1,788
-
02-23-2014, 06:59 PM #18Web Hosting Master
- Join Date
- Jul 2007
- Location
- Virginia
- Posts
- 1,314
-
02-23-2014, 09:05 PM #19Randy
- Join Date
- Aug 2006
- Location
- Ashburn VA, San Diego CA
- Posts
- 4,615
The issue in my case is most likely in the libraries I've been using. Originally written many years ago by AMSIX for their brocade gear (and somewhat sloppily, at that) and never updated since. It's one of those coding styles nobody except the author would touch without starting over. Fortunately we only need to be parsing brocade sflow for the time being but this thread has motivated me to look further .
Fast Serv Networks, LLC | AS29889 | DDOS Protected | Managed Cloud, Streaming, Dedicated Servers, Colo by-the-U
Since 2003 - Ashburn VA + San Diego CA Datacenters
-
02-23-2014, 09:49 PM #20Web Hosting Master
- Join Date
- Jul 2007
- Location
- Virginia
- Posts
- 1,314
Do people export flows from TOR switches or routers? Or both? I don't see why you'd need exports from both since that seems redundant.
Similar Threads
-
Security log analysis tools
By AndyB78 in forum Hosting Security and TechnologyReplies: 2Last Post: 08-04-2011, 10:04 AM -
Which are the best & free tools for doing competitor SEO analysis?
By jiten702 in forum SEO / SEM DiscussionsReplies: 2Last Post: 07-02-2010, 03:50 PM -
Complete Site Analysis - Link Popularity and Domain Analysis
By elsonar in forum Software & Scripts OffersReplies: 0Last Post: 02-15-2009, 02:20 PM -
One all encompassing Online Website Analysis Tools
By jphayes in forum Programming DiscussionReplies: 2Last Post: 07-27-2007, 12:22 PM