This weekend I was on call and took a particularly interesting call. The solution turned out to be a case of what not to do.
My company is in the prospect phase with this client and was not involved until the going got rough. On Friday this client changed ISPs and phone systems. They changed the following items: firewall (Sonicwall TZ170 to Cisco ASA 5505), switches (Cisco to Adtran), external IP space (a full /24), and phones.
Everything wasn’t initially configured from the get go. On Friday night the phone pieces were done right, but the ASA wasn’t entirely configured. The companies IT individual was working with an individual from out of state to configure it. The local IT individual is a web developer and has limited experience working in the networking stack. Eventually, approximately 02:00 or 03:00 Saturday morning they finished up, but his primary web server (Windows Server 2003) which handles e-commerce was not working. He went back in the next day, during the first snowstorm of the year, and was unable to get the server to properly serve web pages or authenticate against his LDAP server, an OS X Open Directory implementation. He contacted our 24/7 helpdesk and the call was escalated to me.
After trying to troubleshoot the issue for 30 minutes over the phone and not having remote access to any of the equipment onsite or the ability to grant myself access – I did not have access to the ASA – I drove to the clients location in the storm.
What I came across was a very simple LAN – a single /24 for PCs/Servers and a separate /24 voice VLAN – but extremely odd behavior. The web server could communicate outbound, it could ping it’s gateway, and it could ping itself. If I tried to ping the LDAP server it would time out and no ARP entry would be created. When I tried to ping it from the LDAP server, no pings were transmitted, but an ARP entry would appear for it.
I reviewed the switches for errors and the ASA configuration, although everything was indicating a clear issue at Layer 2. I escalated to our Level 3 technician for more indepth review of the ASA as I am still fairly new to that product line. While he dug further in the ASA and noticed some odd behavior, I check the local routing table of the problem web server.
I found a persistent route that was exactly the same as the local gateway. I’ve never seen a static route set to point to the local gateway before. Upon removing the static route, everything began to work. I could ping the LDAP server, my ARP table looked right, and the websites were serving properly.
This is a clear example of what not to do or if you are collapsing your network, verify that you’ve removed static routes.
It was an interesting problem to solve and something I’ll certainly keep in the back of my mind the next time I come across odd behavior.
