Each WISP has their own method of monitoring the network, making sure everything's okay.

[June 13, 2006]

On the ISP-Wireless list in June, KC asked

What do you all use to keep track of what is going on on your network?

[MS replied] "MRTG mostly. We'll start to run a system that uses some of the netflow data available to us this summer. Mostly though, we listen very closely to our customers!"

[BW replied] "nagios...with some scripts that do change control...sort of"

[RS replied] "nmis. It's a wonderful thing. The sample screens shown there don't do justice to the "large" dashboard. The metric # on my network for 4 yrs now is 94.567 Considering I monitor every customer unit, that's a pretty good number!"

[RT replied] "Whats Up Professional—and you can tweak the ICMP timeout, retries, etc."

[RS replied] "Anyone tried Zabbix? We currently use nagios/cacti and some custom stuff, but are always looking at alternatives."

[ST replied] "Nagios and Cacti. Although I am still in search of some decent netflow tools."

[TR added] "Don't forget Mirotik's new DUDE."

[BW replied] "My issue with dude is that I'd have to populate it from scratch. :("

DH had built a system:

"I could never could find anything that I could control to tweak it to my liking (and there were time-outs, time lapse, false alarms, etc.). So I grew my own poor man's monitor using Perl and its built in ICMP module. You can grow this to anything you like. I've even incorporated a module that e-mails the failure notices out. You can tweak it to ping once, then try again a couple of times to make sure that the device you are trying to monitor is indeed down.

The Perl part is free from ActiveState, and you can ask for the script. The other monitoring as far as bandwidth usage, I use MRTG with SNMP based connections to all of my switches, routers and SNMP-based equipment."

NM had also built a system:

"We have some home-grown Perl scripts that ping our CPEs every 2 minutes, and then dump the results (IP, timestamp, %successful) into a MySQL database. We do the same thing for SNMP monitoring for bandwidth usage on our routers and APs. We can then pull the data out of MySQL and into PHP+JPGraph and create pretty little graphs for customers. Also helps to pull up a specific time of the day (or several days back). It's part of a custom built administration interface (customers, billing, monitoring, accounting, dialup, etc) that I've been slowly building over the last few months."

ST had a related question:

What software have you looked at handle your netflow data? I'm capturing it currently from my edge mikrotik and saving it with nfcapd which is part of nfdump (nfdump.sourceforge.net) but I haven't found anything to graph the data in a manner I like yet. I haven't look at too many packages yet.

[RS replied] "We run ntop, but it's still not real intuitive."

JC summarized the discussion, concluding:

"The answer to this question really kind of depends on what you mean by the phrase 'going on'. I have seen two types of environments, both with their own needs and corresponding tools. One or both of these scenarios may apply to your situation, but depending on the operational style of your company, you will probably find one set of tools more useful than the other.

The first scenario is that of a service provider. This is where people will recommend tools such as Nagios, WhatsUp, Cacti, OpenNMS, MicroMuse, etc. In this particular situation, you don't particularly care about what traffic is passing over the network, but rather how much and with what sort of reliability. All of the information that you care about is at the lower levels of the network, typically only looking so high as the availability of a TCP/IP Layer 4 service. I think that this is what you are looking for.

The second type of situation is much more controlled in that you are looking at the types and destinations of traffic. This is where the netflow data comes in, in addition to tools like Ntop. With these, you can watch the characteristics of your traffic, perhaps slapping the hand of those who do "bad" things to the network, etc. This is looking at the upper few layers of the protocol stack, looking at the actual data that travels across a network. This particular approach assumes, as in any other good layered model, that the lower layers are working correctly.

The ideal solution, in my mind, is a tool such as Cacti to trend the number of frames traveling across a given Layer 2 entity, something like Nagios to check for availability of Layer 3 devices and Layer 4 services, and then netflow analysis to track the Layer 5 element.

You are working with a layered protocol stack, and it makes sense to me to use a layered approach to analyze a layered stack. However, the downside to this approach is that you end up with using 2-5 separate tools to monitor your network, while most people, myself included, would prefer a single tool to enter devices into, and a single place to look for problems.

On balance, I would rather have 50 tools that all did a single job effectively than a single tool that did 50 things in a mediocre fashion. This is true much in the same way that no matter how good your multi-tool of choice gets, it will never fully replace a properly stocked toolbox. (See also: Unix Design Philosophy)

—End