The Joys of Verio Tech Support

What a day! When I got in to the office this morning I had a call from a client that their web site seemed to be down. I checked the site from my computer which is on the same class C and could see it without any problem. However, from beyond our router the site couldn’t be seen. We were in the process of moving this particular client’s sites to a new server so I assumed I must have hosed up some config files on the server. The server was running Redhat 6.2 and had 5 IP addresses configured, from .80 to .85 on our C block. It turned out that packets going to or from those address could not pass from our network onto The Internet. I could ping the outside world from my workstation at .153 or from our name server at .2 but not from any of the IPs on the clients machine even though they’re all in the same class C. Now, the weird part was that I could ping the client’s machine from inside the class C.

This had me baffled for a while. What could suddenly make several IPs in the middle of a Class C block stop working? Nothing in the configuration of the router had changed during the last 24 hours but the logs clearly showed that those IPs ceased to be routed at about 2am last night. It had to be something on the server itself, I decided. I completely reconfigured the networking from scratch several times with no results. Then, just to make sure I wasn’t going insane, I tried configuring it with a completely different IP address. It worked! So, it wasn’t the server at all, there really was a hole in our class C. I started checking logs on other servers and determined the problem was much more widespread than just the one server. It turned out that about 60 of our IPs had gone dead and quite a few of those were customer sites.

Then it hit me. It had to be the router on the upstream side of the T1 at Verio. Only Verio could screw something up this bad. Unfortunately, it was afternoon by now and all the local Verio people I knew were gone for the day. The only thing left was to call the Verio support center. This is not something I enjoy. Everyone hates having to talk to tech support but Verio’s level 1 makes most tech support people seem like geniuses. I believe Verio requires each applicant for a position in tech support to demonstrate complete lack of knowledge in a wide range of fields including The Internet, routers, TCP/IP, and protocols. At the same time, they have to demonstrate an absence of common sense and an inability to spell or remember proper names. Having english as a second language and the ability to speak way too quietly to hear are also valuable skills you’ll need to work in Verio level 1 tech support. But I digress…

I called, waited on hold, and eventually got someone on the phone. After a lengthy conversation in which I spelled my name and the company name several times and assured them that, yes, I really was one of their customers, I got to explain my problem. After just a few repititions, he was ready to go to work on it. The conversation went something like this: (but to be totally realistic, insert a line after each statement made by Verio in which I asked him to speak up so I can hear what the heck he’s saying.)

Verio: If you’ll give me your IP address, I’ll check and see if it’s working.
Me: I already know some of the IPs aren’t working, that’s why I called.
Verio: What’s your IP address?
Me: There is a range of IPs in one of our C blocks that seems to be dead, do you want one of the IPs that is working or one that isn’t?
Verio: I will ping your IP and see if it’s working, what is it?
Me: is one of the IPs that’s dead, is that what you want?
Verio: Hmmm… that IP doesn’t seem to be working.
Me: Yes, that’s why I called.
Verio: Do you have more than one IP?
Me: Yes, we have several C blocks, but only one is experiencing routing trouble.
Verio: What’s the IP address that has trouble?
Me: The one you just pinged is experiencing trouble, do you want another one?
Verio:How many IPs do you have?
Me: We have several C blocks, but only one is experiencing problems
Verio: 1 C block? How many IP addresses are in that block?
Me: 255, the same as any C block.
Verio: What’s the range of IPs in your C block?
Me: All of the IPs in the C block are ours.
Verio: What are IP numbers in your C block?
Me: They range from 0-254
Verio: What’s the IP of your router?
Verio: Ok, I can ping your router, so the problem must be at your end, is there anything else I can do for you today?

At this point, I tried again from the top and explained that we had “many IP addresses” and some of them had suddenly gone dead. I explained that this was due to a routing problem in either my router or their router and that since I had already checked my router, their’s was the likely source of the problem. At this point, he asked for the password to our router so he could check and see if it was the source of the problem. I repeated my previous sentence and he seemed very confused but promised he would “look into it” and call me back, possibly by monday. I explained that we had customers who were down, and the problem needed to be fixed immediately. He promised they’d look into it right away and gave me a case number but wasn’t sure when they would be able to call me back.

I hung up and played a round of Robotron in the game room to vent some frustration. While I was comtemplating the problem, I suddenly realized what Verio had done. The exact number of IPs that had gone bad was mostly likely 64 and the reason had to be that someone at Verio had created a subnet out of our class C and assigned it to some other customer. It was about an hour later now and I called Verio back. This time I got someone who actually spoke English and talked like a regular human. I explained to her that I knew exactly what Verio had done and got her to type my theory into the notes on the case history. I explained again how many of our customer sites were down and how important it was that the problem was fixed tonight. She elevated the problem to level 2 tech support, who she promised would call me shortly.

After about an hour I called back and asked what happened to the level 2 call. She checked the case and said they had assigned someone to work on it and had verified that my theory was correct – they had accidently taken 64 of our IPs, and assigned them to a DSL customer last night. After another hour I got a call saying they’d put in a static route as a temporary fix and would have a permanent fix by morning. Sure enough, we were up again. They said this was very unusual and probably wouldn’t happen again. Great, I’ll add that to the list of major screw-ups that probably won’t happen again. Like their mishandling of secondary DNS support for our domains, or the alleged fiber cut back in November, or the comedy of errors we went through getting the T1 installed in the first place (there’s enough material there to write a whole book) and there was the infamous Verio Focus Group experience too. Well, enough ranting for today.

Titan A.E.

I’m posting this from the Mozilla M16 release. It’s still got a long way to go before it’s ready for everyday use but it’s slowly getting there. It still doesn’t seem to work at all on sites that use dynamic html. But M16 is the first version that has rendered the table-based layouts on Slashdot and Freshmeat in a way that seems almost correct.

Nothing too exciting to report over the weekend. It did stop raining long enough to mow the yard (it’s raining again right now, in fact). We got the newly released Philip Glass CD, Symphony No. 3. Highly recommended! It also includes a couple of shorter pieces, one of which is written in the chromatic scale – a bit of a departure from the norm for Glass and definitely worth a listen.

Susan and I went to see Titan A.E. this weekend. As I’m sure everyone is aware by now, the movie sucked. After all of the hype that it was going to be an animated movie targeted at adults instead of children, I was a bit suprised to find that just the opposite seemed to be true. Everything in the movie seemed targeted at very young children. All the characters were drawn in Disney-esque, ultra-cute style. For the most part they all behaved like characters from a Disney children’s film too. I guess when they said an adult animated film, I was thinking of Heavy Metal or Bakshi’s films like Wizards.

The plot doesn’t help much. Some movies have holes in their plot but this one was all hole. Half the time the story is set up on the premise that the Drej destroyed the Earth because they feared what mankind would do with a powerful new ship it had created called the Titan. The other half of the time, the premise is that humans created the Titan to save humanity because they knew the Drej were planning the Earth’s drestruction. Huh?! So, the Drej destroyed the Earth because Earthmen built the Titan but Earthmen built the Titan because the Drej were going to destory the Earth? Don’t even try to figure it out. The movie switches back and forth between these two contradicting story lines frequently and at unpredicatble times. It’s almost like two movies with different plots were made and edited together into one incomprehensible mess.

Adding to the two-movies-in-one effect is the combination of traditional cell animation and cheap 3D computer graphics. Some scenes look like they’re right out of a traditional Diseny movie and others look like 2D animated characters have been chromo-keyed onto a 3D landscape somebody spent a couple of hours rendereding in Bryce. Mixed in with this are a few higher-quality ray-traced renderings. So, let’s see, that’s two plots full of holes combined into one, goofy combinations of 3D and 2D animation, terminally-cute Disney characters suitable for a childrens movie, stupid dialog, and hmmm… I’m sure I had a longer list of complaints than that. Oh well, you get the idea. The one thing that worries me is that I seem to agree with JonKatz on this one!

Crazy Connector Revealed

A couple of people noticed that I screwed up the filename in the link I included yesterday for mod_virgule. The prize goes to fozbaca for being the first to email me and report the problem. I fixed the link, so anyone who had problems grabbing my new and improved (and unofficial) version of mod_virgule should try again.

Graydon on advogato mentioned that he’s seen an increased level of Spam lately. I’ve noticed it too. Incoming spam at our domains has increased several-fold over the last week or two. I usually manage to get one or two spammer throw-away accounts shut down per week and we use ORBS and RBL. In spite of all that, the spam levels continue to rise.

I finally got my cable assembly for the Kyocera LCD panel working. I was a little worried that it wouldn’t work at all – I’m no expert when it comes to the use of a soldering iron and there are 27mhz signals going through my completely unshielded mess of wiring and a cheap perf-board. But it actually does work! I was so impressed I got Randy to snap a few pictures of it. If you’re bored, have a look:

[[popup:cable1.jpg:cable1-small.jpg:cable adapter:1:center]]

[[popup:cable2.jpg:cable2-small.jpg:cable adapter:1:center]]

Above are twol pics of the perf-board that connects the two ribbon cables. The blue 50 pin cable goes to a mezzanine LCD controller in the Ziatech box. The grey 31 pin cable goes to the Kyocera LCD panel. The little black pot at the center of the perf-board adjusts the LCD contrast.

[[popup:cable3.jpg:cable3-small.jpg:hardware setup:1:center]]

Above is the whole setup. The Black box is the Ziatech STD100 unit, the tiny green board on far right is an inverter that provides 1200vac for the LCD backlight, and that little board to the left of the LCD panel is the touch-screen controller.

[[popup:cable4.jpg:cable4-small.jpg:Hardware setup:1:center]]

The whole setup from a different angle. To give you an idea of the scale, the LCD panel is 7.2″ diagonal.

[[popup:cable5.jpg:cable5-small.jpg:Me working on hardware:1:center]]

And that’s me messing with the hardware.

Melody Explorer

I was thinking about ways of generating music mathematically last night. I’ve had this idea for a while of making a compositional tool that works something like the old Kai’s Power Tools Texture Explorer did. You could generate a sequence of notes, then mutate the rhythm, melody, and harmony over time using genetic algorithms to approximate what the user wants. Anyway, the issue that occurred to me was that we usually do our math in base-10 but music generally uses either one of the common 7 tone scales such as the diatonic or the 12 tone chromatic scale. One possibility is to work in a 10 tone microtonal scale but a lot of people wouldn’t find the sound very pleasing (microtonal scales usually have more tones than the chromatic scale anyhow). The other option, of course, is to let the user pick the scale and do the math in an appropriate base for the scale. It sounds like fun but I probably won’t have the time to do it anytime soon as it’s not the sort of project someone is likely to pay me for! Oh well.

Speaking paying work, I’ve been working on a new web site that’s going to be using mod_virgule and have been making loads of changes to the code. Some bug fixes and some new features. Raph‘s been way too busy lately to do much maintenance on the official release, so I’ve decided to release my patched version for anyone who’d like to have a look at it. Once the official version starts seeing active development again, my version will probably mutate back towards the main source tree and go away.

Crazy Connector Cables

I’ve spent the last two days building a cable to connect a 7.2″ Kyocera color LCD panel to a Ziatech controller. The Ziatech connector is a 50 pin ribbon cable and the Kyocera is a 31 pin cable. Some pins on one side have multiple connection on the other and some of the connections (like contrast) require variable voltages that have to be obtained by inserting potentiometers inline. The result so far is a rats nest of wiring. Fortunately, a special adapter board is being designed for the final product that the two different ribbon cables can just snap into. Then there’s the inverter that powers the backlight. 12v DC in, 1200v AC out – don’t touch it. Well, it’s a change from software anyway.

Speaking of software, I sent Raph another load of patches for mod_virgule including a fix to prevent a nifty little DoS attack I discovered this week and changes to the render functions to make them at least approximate legal HTML. Sounds like Raph’s got his hands full though, so I don’t know when we’ll see any of them integrated (maybe it’s time to pass the code on to another maintainer so development can continue?).

Texas Scottish Festival and Highland Game

Saturday we went to the Texas Scottish Festival and Highland Games. After going last year, I decided this was something we should do every year. There is always lots of cool stuff to see but it’s really the music that makes it worth the trip. The Houston group, Clandestine, was there again this year and sounding as good as I remembered. I picked up their newest CD. Highly recommended. I also picked up two of Ed Miller‘s CDs. We spent most of the day there and at least half of it was just enjoying the music. But we took time to wander around looking at the other sights: sheepdogs, bagpipes, Scottish harps, shortbread, and lots of vendors selling all sorts of strange things. We missed the Scotch whiskey tasting seminar though – maybe next year.

I packed up my latest mod_virgule patches for Raph this morning, maybe we’ll see some new code in CVS soon.