Verio – Just Say No

Posted on 20 February, 2001 by steevithak

Verio tech support… how can any group of people be so amazingly, mind-numingly, unbelievably incapable of providing support for technical things at Verio or elsewhere. :-(

I’m afraid my mind is still reeling from trying comprehend the massive stupidity and mismanagement involved so I can’t bring myself to write out the full details yet. Perhaps when it’s finally over – we’re only 7 days into it right now and it’s a fairly simple problem that any sysadmin with some BIND experience could fix in half an hour, so I’m betting we’re got at least another 7 days to go before someone at Verio blunders into a solution. In the meantime, you can entertain yourself reading the past exploits of Verio tech support from their simplest DNS blunder, and their wonderful work with Worldcom, to the famous focus group event and, my favorite, the story of the first of several times they mis-allocated part of our IP addresses to a DSL customer.

I’d offer the also very interesting story of our 3 month installation saga on our first T1 and the related double billing and mysterious shifting account number problems but by the time I fully regained my sanity from those I’d blocked most of the details from my memory. Suffice to say of the current mess, if Verio ever offers to do your secondary DNS for you, just say NO!

Verio Does the Right Thing

Posted on 29 January, 2001 by steevithak

Since I frequently rant about how bad Verio is, it seems only fair to mention when they occasionally do something good. Verio is the upstream provider for cryptome.org. Verio was recently notified by the MPAA that Cryptome was in violation of the DMCA because it contained a transcript of John Hoy’s declaration from the DeCSS trial in NY. John Hoy is (was?) the president of the DVD CCA and the document in question is part of the public record so the MPAA’s claims seem a bit absurd. (Mr. Hoy just happened to enter the source code to DeCSS into the public court record as part of his declaration and that’s why the MPAA is trying to supress it.) Rather than shutting down the site like most ISPs seem to do when threatened by the MPAA, Verio exchanged a couple of emails with John Young of cryptome, and agreed the site should not be taken down. There’s a cnet article on the events and the entire email exchange is available on cryptome for the curious.

I also ran across an interesting statistic at attrition.org today. They have a nice pie chart showing a breakdown of reported server break-ins by OS type.

Verio Strikes Again

Posted on 24 January, 2001 by steevithak

Verio strikes again. About half the IPs in our C block seemingly went dead the other day. Web servers on those IPs couldn’t be hit from outside and we couldn’t ping anything beyond our own router from them. Something along these lines happened last year and it turned out someone at Verio had decided to subclass our C block and assign part of it to a new DSL customer without bothering to tell us (or even pinging a few of the IPs to see if anyone else might be using them). I called Verio and was able to confirm my theory fairly quicly – it was Verio’s doing. But this time they simply put a bogus static route in one of their routers that mis-routed half our C block for no apparent reason. It was suprisingly easy to get the tech-support person to understand what the problem was too. That’s very unusual as most Verio tech-support personal seem to pride themselves on their total lack of comprehension of anything technical and how to support it. I did have to tell them which router the problem was in but I’m used to that.

Getting it fixed was classic Verio. The tech-support department can’t actually fix anything themselves (and I don’t blame them for that – I wouldn’t let those people near any of my servers either!). They have to email a central problem resolution center and request that a third group be contacted that might actually fix the problem. They claim there is no way for them to contact the second group directly because that group has no telephones. And they claim the third group, the ones that actually fix problems, also have no telephones. Both groups can only be contacted by email. Of course, they have no idea how long or even if the other groups will take in responding to their email. It blows my mind that they actually expect anyone to believe that a corporation the size of Verio has entire departments of people who can’t be contacted by telephone.

From the time I first contacted Verio, it took them around three hours to fix the problem. That’s probably two hours and fifty nine minutes of waiting for people to get the email requesting the fix and one minute for someone to log into the router and remove the bogus route.

Verizon DSL Disaster Continues

Posted on 8 January, 2001 by steevithak

GTE may have changed their name to Verizon but they still have the same really bad customer service. They could give Verio a run for the money when it comes to having the least helpful, most incompetent customer service.

Several months ago I made the mistake of calling Verizon and asking if I could possibly get DSL service. They said it wasn’t available in my area and I thought that was the end of the matter. But Verizon decided to bill me for DSL even though I can’t have it. So I’ve spent several hours on the phone with them since then trying to get the bogus DSL charges removed from my bill. Every month when a new phone bill arrives it turns out that not only have they not fixed the problem, they’ve added a new month of charges.

The general process you go through in a case like this is to call the phone number provided on the phone bill for billing problems and explain the problem (even when you ask them to, they never keep a record of any previous calls, so you always have to start from scratch). Next the billing office will tell you that you have to talk to the Internet office. The Internet office will tell you to call the DSL office. The DSL office will say they can’t get involved in billing problems, they only handle DSL problems. Sometimes if you complain enough and sound angry enough, they forward you to a supervisor who makes up some whacky explanation for the how to get DSL charges off your bill.

These whacky explanations will be different each time you call and, of course, none of them work. Today’s supervisor insisted that I had to get someone in the DSL department to send an email to the accounting department requesting that the non-existent DSL service be removed from the bill. This was because, they said, the accounting department isn’t allowed to correct billing problems discovered by customers, but will instantly fix billing problems discovered by other internal Verizon departments. Yeah right. last months solution involved setting up a three or four person conference call with representatives from all three departments because they had to all simulaneously fix the problem. At one point it was even suggested that the easiest solution would be to “just pay the bill since it wasn’t very much”. I can hardly wait to see what they come up with next month.

The Joys of Verio Tech Support

Posted on 23 June, 2000 by steevithak

What a day! When I got in to the office this morning I had a call from a client that their web site seemed to be down. I checked the site from my computer which is on the same class C and could see it without any problem. However, from beyond our router the site couldn’t be seen. We were in the process of moving this particular client’s sites to a new server so I assumed I must have hosed up some config files on the server. The server was running Redhat 6.2 and had 5 IP addresses configured, from .80 to .85 on our C block. It turned out that packets going to or from those address could not pass from our network onto The Internet. I could ping the outside world from my workstation at .153 or from our name server at .2 but not from any of the IPs on the clients machine even though they’re all in the same class C. Now, the weird part was that I could ping the client’s machine from inside the class C.

This had me baffled for a while. What could suddenly make several IPs in the middle of a Class C block stop working? Nothing in the configuration of the router had changed during the last 24 hours but the logs clearly showed that those IPs ceased to be routed at about 2am last night. It had to be something on the server itself, I decided. I completely reconfigured the networking from scratch several times with no results. Then, just to make sure I wasn’t going insane, I tried configuring it with a completely different IP address. It worked! So, it wasn’t the server at all, there really was a hole in our class C. I started checking logs on other servers and determined the problem was much more widespread than just the one server. It turned out that about 60 of our IPs had gone dead and quite a few of those were customer sites.

Then it hit me. It had to be the router on the upstream side of the T1 at Verio. Only Verio could screw something up this bad. Unfortunately, it was afternoon by now and all the local Verio people I knew were gone for the day. The only thing left was to call the Verio support center. This is not something I enjoy. Everyone hates having to talk to tech support but Verio’s level 1 makes most tech support people seem like geniuses. I believe Verio requires each applicant for a position in tech support to demonstrate complete lack of knowledge in a wide range of fields including The Internet, routers, TCP/IP, and protocols. At the same time, they have to demonstrate an absence of common sense and an inability to spell or remember proper names. Having English as a second language and the ability to speak way too quietly to hear are also valuable skills you’ll need to work in Verio level 1 tech support. But I digress…

I called, waited on hold, and eventually got someone on the phone. After a lengthy conversation in which I spelled my name and the company name several times and assured them that, yes, I really was one of their customers, I got to explain my problem. After just a few repetitions, he was ready to go to work on it. The conversation went something like this: (but to be totally realistic, insert a line after each statement made by Verio in which I asked him to speak up so I can hear what the heck he’s saying.)

Verio: If you’ll give me your IP address, I’ll check and see if it’s working.
Me: I already know some of the IPs aren’t working, that’s why I called.
Verio: What’s your IP address?
Me: There is a range of IPs in one of our C blocks that seems to be dead, do you want one of the IPs that is working or one that isn’t?
Verio: I will ping your IP and see if it’s working, what is it?
Me: xxx.xxx.xxx.84 is one of the IPs that’s dead, is that what you want?
Verio: Hmmm… that IP doesn’t seem to be working.
Me: Yes, that’s why I called.
Verio: Do you have more than one IP?
Me: Yes, we have several C blocks, but only one is experiencing routing trouble.
Verio: What’s the IP address that has trouble?
Me: The one you just pinged is experiencing trouble, do you want another one?
Verio:How many IPs do you have?
Me: We have several C blocks, but only one is experiencing problems
Verio: 1 C block? How many IP addresses are in that block?
Me: 255, the same as any C block.
Verio: What’s the range of IPs in your C block?
Me: All of the IPs in the C block are ours.
Verio: What are IP numbers in your C block?
Me: They range from 0-254
Verio: What’s the IP of your router?
Me: xxx.xxx.xxx.1
Verio: Ok, I can ping your router, so the problem must be at your end, is there anything else I can do for you today?

At this point, I tried again from the top and explained that we had “many IP addresses” and some of them had suddenly gone dead. I explained that this was due to a routing problem in either my router or their router and that since I had already checked my router, their’s was the likely source of the problem. At this point, he asked for the password to our router so he could check and see if it was the source of the problem. I repeated my previous sentence and he seemed very confused but promised he would “look into it” and call me back, possibly by Monday. I explained that we had customers who were down, and the problem needed to be fixed immediately. He promised they’d look into it right away and gave me a case number but wasn’t sure when they would be able to call me back.

I hung up and played a round of Robotron in the game room to vent some frustration. While I was contemplating the problem, I suddenly realized what Verio had done. The exact number of IPs that had gone bad was mostly likely 64 and the reason had to be that someone at Verio had created a subnet out of our class C and assigned it to some other customer. It was about an hour later now and I called Verio back. This time I got someone who actually spoke English and talked like a regular human. I explained to her that I knew exactly what Verio had done and got her to type my theory into the notes on the case history. I explained again how many of our customer sites were down and how important it was that the problem was fixed tonight. She elevated the problem to level 2 tech support, who she promised would call me shortly.

After about an hour I called back and asked what happened to the level 2 call. She checked the case and said they had assigned someone to work on it and had verified that my theory was correct – they had accidentally taken 64 of our IPs, and assigned them to a DSL customer last night. After another hour I got a call saying they’d put in a static route as a temporary fix and would have a permanent fix by morning. Sure enough, we were up again. They said this was very unusual and probably wouldn’t happen again. Great, I’ll add that to the list of major screw-ups that probably won’t happen again. Like their mishandling of secondary DNS support for our domains, or the alleged fiber cut back in November, or the comedy of errors we went through getting the T1 installed in the first place (there’s enough material there to write a whole book) and there was the infamous Verio Focus Group experience too. Well, enough ranting for today.

One Year of Weblog Provided by Newslog

Posted on 5 May, 2000 by steevithak

Today marks the 1st anniversary of my on-line news postings. I posted the first of these more-or-less daily news updates on May 5, 1999. That’s a lot of old news to read. Since I hacked some syndication code into newslog in early March, these news entries have been echoed to Advogato as well. I expected most of the embedded links in my old entries would have gone bad by now but a suprising number of them still work. Now, let’s see if I can keep it up for another year.

I had an interesting experience Wednesday. I got picked to be in a focus group doing some marketing research for Verio. If you’ve been following my exciting news regularly, you may remember that we’ve had T1 downtime problems, DNS support problems, dupicate billing problems, and even dial-up problems with Verio in the past. When I got called to take part in a Verio focus group, I figured it would be a great chance to tell them what I thought of them. Turned out that of 10 people in the focus group, all but one had come with a similar history of catastrophically bad service and a similar desire to tell everyone what they thought. Almost everyone there had been a customer of Onramp.net prior to the Verio buyout and without exception we all rated Onramp as one the best ISPs we’d every used and Verio as the worst. I don’t know if they found out anything useful from us be we enjoyed getting paid to sit around sharing Verio horror stories for couple of hours.

And I’m sure everyone would like to hear about my latest car repairs. ;-) As I pulled out of the driveway yesterday morning, my muffler broke loose on one side and started dragging behind the car making horrible scraping sounds. It had apparently become so rusted that the mounting hardware broke. This is the Midas lifetime warranty muffler that I got to the replace the rusted-out Acura muffler I bought sometime back to replace the rusted-out free replacement muffler installed by the Acura dealer to replace the rusted-out factory muffler. Well, I figured with my lifetime warranty and Midas just down the street, it would be no big deal. So, I drove a couple of blocks with the completely unmuffled engine sounding like an Indy race car (combined with the lovely sound of metal scraping on asphault). When I got to Midas, I learned that while the muffler itself has a lifetime warranty, I have to pay for labor, mounting hardware, a large assortment of exhaust pipes, and a variety of other stuff totaling about $200. And to make things worse, I had to wait in a room filled with nothing but Sports Illustrated magazines and a TV showing an old episode of Knight Rider. Fun.