Best of the ISP-Lists

What Would Cause 50% Downtime?

When companies are advertising four and five nines availability, what problem would cause regular, night-long service outages? Members of the ISP-Wireless list discuss.

[April 5, 2001]

On the ISP-Wireless list in March, ME exclaimed,

"Help! I have a customer that goes down every night about 8pm and does not come back up until 10 or 11 the next morning. What would cause that?"

Some respondents suggested looking at the effect of changes in temperature:

[MF observed] "Water transitioning between states can cause problems like this. For instance, from humidity/vapor to water or ice as it cools into the night, and back when it warms the next morning. Take a look at all of the equipment that is exposed to a wide temperature change."

[RB added] "Also look to make sure that there's some cable slack before all connections, i.e. don't go straight up the tower and straight into the amp or antenna. Slight bits of expansion and contractions of the cable can stress a connection."

Others recommended investigating personnel shifts that occur at the relevant times of day:

[RR recalled] "I had a similar problem: it finally turned out to be a bad hub. When they turned off the air conditioning at quitting time it would overheat, until the air conditioning came back on the next morning!"

[KK added] "Is the CPE on a lighting circuit that might be on a timer, or near where an employee could turn it off or on by a switch that perhaps isn't immediately obvious? I'd look at the employees first."

[MS agreed] "It's gotta be some kind of business cycle: something in the path is changing at those times."

MH offered a somewhat more obscure possibility:

"And the pirate radio tower is how far away? We had a Christian AM radio station that would crank up the power at night and bled on phone lines, Ethernet, radios: it affected everything, because he was not really equipped for putting out the power levels he was using."

JL countered with a systematic way to break down the problem:

"Get the details. If the problem is real, the next step is to divide and conquer. Is the problem at the customer's destination, at the ISP's end, at the access point, at the CPE, at the subscriber's equipment, or due to external influences? All the above can be tested with either measurements or replacements.

From the top:

Replace the destination. Is the entire system dead, or are only certain destinations dead? Can you ping by IP address?

• At the ISP, does the customer show up in the RADIUS logs, wtmp, syslog, or other diagnostics? Any error messages (login failures, duplicate IPs, etc.?)
• It will not be possible to replace the RF parts and pieces. Therefore, make measurements. If your unspecified hardware has diagnostics or SNMP monitoring, measure the signal levels at the access point or CPE radios. Are they marginal? Any retrans? Fixed speed or adjustable with signal quality? When the link is up, try to do a performance or bit error rate test to determine link quality. If there are external or environmental influences, they will show up in signal or link quality measurements.
• Is it only this customer? Are nearby wireless customers having similar problems? If it's only one, then whatever is happening is probably at the customer's site.
• If the customer has multiple computers, have them try a different machine. If they have a home router/NAT, have them check the configuration. Try a known working machine plugged directly into the wireless bridge without the router.

Eventually, the location of the failure will become evident, which will reduce the witch-hunting and finger-pointing to a manageable minimum."

—End

     

Related articles:
	[Apr. 4, 2001]	According to Whom?
	[Nov. 11, 2000]	IP Conflict
	[Jul. 7, 2000]	Should I Accept This?