Redundancy across ISP’s

One of the first projects I was given when starting this new job was to provide Internet redundancy to a customer.  They already had Internet access via ISP #1 (names change to protect the idiots).  They contacted ISP #2 and requested high-speed access and a block of public IP address.  While they were doing this, an Adtran NetVanta router appeared with 2 Internet connections and 8 LAN ports.  The idea was simple, replace the Cisco device at the company with this Adtran router, connect both ISP’s into the new router and provide redundancy.  This is how the project was presented to me.

Since I was new, and did not know much about Adtran capabilities, I said sure, and started down a long and frustrating path.  Problems with monitoring the active connection, the inability to set the “Active connection” was just the start of it.  As I moved past those issues, new issue cropped up, DNS round-robin was a foreign concept to the DNS hosting company, then there were Point to Point VPN connections that had to be created for both connections.  P2P VPN was an issue as the other end of the VPN would not accept multiple Peers to initiate the tunnel.  Then there were 4 client to site VPN connections to worry about.  

Engineering around the P2P tunnels was straightforward..  A simple ACL on the outbound ISP #2 connection would NAT the traffic so it looked like it was coming from another IP address.   VPN tunnels working.  Then the Client VPN issue was ready to be tackled.  All I had to do was un-install the Cisco software and install the VPN client from Adtran………….

3 weeks later, the VPN software does not work with Trend Micro Anti-Virus,   time to look at the Cisco hardware already in place.  ASA 5505 with Security Plus license, bingo…

3 months of dealing with problem after problem with the Adtran hardware, 3 hours and redundancy is set up on the Cisco device.  If ISP #1 goes down, ISP #2 takes over, since the 2 connections share the same NAT pool, it takes less than 2 seconds to fail-over, and the clients don’t even realize they have switched connections. 

All that is left is to have the 2nd peer address added at the other end of the P2P tunnels and this will be complete.

Long Time, No Update

I know, I know……It has been quitre a while since this has been updated.  I am not dead, yet .  I do however have a long fight with an Adtran NetVanta router that will be posted here.

On-Site….

The vast majority of my work is done at the customer site.  It has been an interesting twist from being the person who brought the “help” in, to becoming the help.  I am currently at my “problem child” customer, and wondering how this entire situation could have been avoided. 

Now, understand, this situation was created before I was even hired into this job, and it was just dumped into my lap.  Basically the customer wants Internet redundancy.  Sounds simple enough to them, and we agreed to make this happen.  the problem is that no one on either side of the discussion sat down to actually compile a list of what “redundancy” means.  So today’s class is on what “redundancy” actually means.

To me, being in networking for the past 18 years, I start the redundancy discussion on the far end of the scale.  This covers all aspects of making sure that a companies web presence (SMTP, HTTP, FTP, DNS, inbound and outbound) are covered.  A sample list is

• Who hosts DNS, will they add secondary and Alias records into your zones.
• Who hosts the MX records and how does mail flow into the organization.
• What incoming traffic is there to account for (HTTP, HTTPS, FTP, ETC)
• Do you want outbound traffic to automatically get re-directed if 1 interface goes down.
• How much are you willing to spend
• What problems have you had in the past year with the current provider

So, here I am, without any questions like this getting asked, trying to setup a brand new Router with multiple interfaces to multiple ISP’s.  My first problem is that there are existing VPN connections that were not identified (2 of them), so now I need to re-create 2 existing interfaces, and add 2 more VPN tunnels for redundancy.    The device I am working with (new router) is not  from a 5 letter company located in Cali…..I have been on the phone with this router’s tech support a lot translating Cisco speak into their speak. 

After about 15 hours of working on getting this configuration setup, I am now mere moments away from trying 1 of the new VPN tunnels.   As Lou Holtz would say on College Gameday “Not so fast my friend”  seems that the other end of the VPN tunnel is not as ready as he said he would be…..Need to reschedule for later in the day…….

ARGH…………………………………..

New Customer/Bad Son

Today, I actually brought a new customer into the store (so to speak). He was actually referred by another customer I had done some work for a few weeks ago. His complaint sounded simple enough, I have a PC that won’t boot. All I get is “Disk Read Failure, Press Ctrl+Alt+Del to Continue”. Sounded like a job for Frank the hardware guy in my head. Haven’t needed to deal with Frank these past couple of weeks, as most of my work has been networking, or simple setup/migrations.

Frank came out of the box on fire. Lets see what is not plugged in, who cracked this box and didn’t mention it, where did I put that damn screwdriver. Calm down Frank…… The system in question was an 18 month old Dell Inspiron desktop, a quick check showed that the warranty was still valid. Sorry Frank, put the screwdriver away.

A quick call to Dell and we start the diagnostic process, of course, every quick test passes. I thnk move on to the time consuming tests. Well about 90 minutes later, all comprehensive tests have passed, and the “owner” mentions that just before the problem started, they removed PGP WholeDisk Encryption.

Uh OH……………………………………………….

Quick question about how PGP was removed, and not a good answer, they just clicked what the “Owners” Son told them to click and then 24 hours later, the system won’t boot. Sure enough, I had a bare metal drive with a fully loaded MBR. A 30 minute restore of Vista Home they now have…..

A working computer

A good lesson in the importance of backing up more than once a year

The knowledge that not everyone who works with computers can fix computer.

And a new Service Provider. I think the $400 bill was going to be taken out of his sons allowance.

ISV Issues

One of the on-going challenges in the MSP space that I have to deal with is that our customers are squarely in the SMB space.  This means that for the most part IT budgets are very tight.  They know that they are already paying by the hour when I show up, so a lot of times this means that the other dollars available are reduced by my rate.
This means that I do a lot of research and use a lot of the applications and utilities that are provided by small Independent Software Vendors.  While these ISV’s are generally very good, and produce some excellent software, it also means that I am not able to get on the spot phone support.
Again, I am billing by the hour, so if I get to a roadblock and the only option is to send an email to the ISV, then I am guaranteeing the customer another hour of billing for when I have to come back to finally resolve the problem.
What are your experiences with ISV shops, and for those of you that outsource your IT, how does if feel knowing that you are going to get that extra time charge?  2 different arguments in my head over this;

Billable time is important to my boss

I am not able to fix the problem in 1 visit, and sometimes we won’t bill the second visit.

Seems like just another headache coming on.

Client Sites

Part of working with a Managed Service Provider is actually getting out of the office and visiting the clients at their site.  Clients to a MSP can come in all shapes and sizes.  So far I have been to a “corporate” site, the cubicle jungle, the quiet that is very loud, the kind of place most IT professionals are accustomed to.  I have also been to a client that you would never think of in terms of technology.  A little double-wide trailer way out in the country.

The challenge in the MSP space is that both of these customers are the same.  Data is data, and to the customer their data is critical.  I personally feel that I am technology agnostic, my wife may not agree, but I don’t care if you are using a Win-Tel platform or an Apple platform.  Windows, Linux, or Mac/OS, it doesn’t matter to me.

The MSP role in the world of small and medium sized business is truly that of PROVIDER.  I have the expertise that these customers need right now.  Once I am done, then they don’t need the expertise any more, and I am off to the next site.

This weeks voice has no name yet, but it is that nagging afterthought of……”Did you provide enough documentation to both the customer, and the next tech that may have to make a call”.  Lets face it, documenting the work done is tedious, boring, and always thought of last, but while you may not like doing it.  It may very well be the most important skill for a tech working for a Managed Service Provider.

Another Turn Around

On Monday the 13th, I started a new job with a managed service provider in my area. They provided any and all IT related services and support to small and mid sized business in the Mid-Atlantic area. I have come on board as a senior network/systems tech and am very excited about it. Not only is the position better than my last one, but I am back in an environment where I get to deal with everything from pulling cable all the way to high end networks, servers, and leading edge technology. This will also start exposing me to some different security work and tools. This week alone I have had to deal with your run of the mill trojans/virus, an emergency restore from backup for 1 companies AP/AR system, the joy of having to scrub up to deal with issues in a hospital pharmacy, as well as why a VPN tunnel was up and running, but the data was not getting to the tunnel. The VPN issue was due to a Barracuda Web filter. The voices in my head were yelling at me when I finally figured it out, that I should have know 2 hours sooner, and the dispatch at the new company was impressed it only took 3 hours to figure it out.

Job Search Update

First, let me apologize for the delay in updates, I have found that I needed to focus full time on my job search.  This means that I was not sleeping in, nor was I focusing on getting those things done around the house that are normally handled on the weekends.  For me, I considered job search as my current occupation. 

The short update is that I have found a new job.  I will be a Senior Tech with a Manager Service Provider in the area, and am looking forward to both starting a new job, and getting the focus of the voices in my head back to technology.

The nice thing about this new job is that I will be focusing on a much broader range of IT services (not just Windows Servers).  So this means that as we move forward, you will get introduced to more of the voices in my head.

Thank you to all who have wished me luck in my job search, as well as a thank you to those of you who actively assisted me in my job search.

Job Search – Progress??

It is a holiday weekend here in the states, and also a slow time to be actively seeking a new job.  It is during these times, that I have to remind myself that I have to be patient.  What I have been doing is making sure that my resume is as up to date as possible, as well as reaching out to my network.  I have always said the IT is a small world, I have met people years ago that I am now hearing from with job leads or just a friendly reminder that this too shall pass.

Just yesterday, I met with a recruiter and as we were talking we found out that I grew up about 20 miles from where she did, and we both worked for Computer Associates.  Now, even though there was a 10 year difference between us, suddenly there is that personal connection.

Now, I have found the vast majority of staffing firms and recruiters to ALWAYS have the best interests of the candidate in mind, that with the current job market it is possible to sometimes slip through a crack, and having that personal connection will sometimes help prevent that.

I am also collecting a list of the websites and strategies that I am finding to be most useful.  While I hope my job search is a short one, I also know that I am not alone.

Changing Focus

When I setup this blog, it was going to be a place to let you follow me as I started a new job, what I was looking to get done, how I was evaluating systems, etc…All the fun stuff you think about during the excitement of starting something new.

Well…it is now time to change focus.  Why?  Well, the economy, do I really need to say anymore….

For the time being, this blog will follow me as I search for a new job.  How I am looking, and the value of relationships outside of this electronic realm. 

There a pleanty of articles, blogs, and newsletters on what to do when you get laid off, so I am not going to attempt to provide any information on that, but what I am going to do is let you follow along on my journey (hopefully very short) for finding a new job. 

Please wish me luck and thanks for following.