Brian's Blog

As far as Christmas celebrations go this year was typical for us. Two fun and exhausting days where the kids get overloaded with excitement and my wife and I get less sleep that we would have on a typical day during the work week. All in all it was a good time with family and the presents were mostly practical for the adults which is what I wanted anyway.

My extra fun was on Dec 23rd just before the holiday weekend. I was thinking it was going to be a nice easy day. I was scheduled to install a new PC for someone around 10AM but instead was redirected to one of our largest customers, a multi-branch local bank. There really isn't much to the story to be told really but keep in mind that even though it's short it was highly stress inducing.

The problem specifically was that every branch was cut off from the main branch. Meaning they could not access any of their files located on the server there. Nor could their database function without the links being up 100%. The odd thing though was that I could access the Internet, ping all the servers across the VPN for all the other branches, and even PCAnywhere to them. What I couldn't do was access a SMB share or see anything other than local LAN systems in the browse list. It would have made 100% perfect sense if I couldn't ping, PCAnywhere, etc but this seemed to target just SMB related things. We had a case open with Microsoft support recently for an issue that was close enough that I could get help without started a new case so I called them up and we checked out the server. It turned out that, for some odd reason, ports 135, 139, and 445 were being blocked and it appeared that Windows wasn't the culprit this time. More detective work showed me that this was the case at all branches that used Choice One as their managed VPN provider but thing were perfectly fine at branches that didn't use Choice One. The next step was clear, and that was to call Choice One up and see what the heck was going on.

Once I slogged through the menus and got an actual person (located in the USA no less!) he didn't see anything wrong, no open tickets, no troubles reported, and no notes related to recent security related changes. Still, I had already proven that I couldn't telnet to any of the SMB related ports, and Windows looked OK so I asked him to check the ports instead of trusting the account notes. Besides, if the OS was the problem we would likely have had local problems as well and been only limited to effected servers. He opened up the ports and in a flash things were working as before.

It turns out that over the past few months Choice One had been closing off the SMB related ports for their clients on externally accessible addresses to combat the spread of viruses. It seems someone took it upon him/herself to close off these ports on the banks VPN stopping it on the internal side and then didn't make any notes in the case related to it. And in doing so managed in a few moments to bring every branch to a halt.

Really, it was such a simple problem at the root of it all, but it took on the order of 5 hours to pin point he problem, get it fixed and then tested. All the while having the manager freaking out and thinking it was some how our fault. Anyway, at least it worked out. The bank was up by the end of the day and it wasn't caused by anything we or anyone at the bank did wrong. Even so, thanks goes to Choice One for poor documentation, lack of procedure, and inducing a mountain of stress just prior to a holiday due to some individuals incompetence.