I guess that he might have been technically correct, after all, the hard drive was replaced with a new one.  However, there was no operating system, no scanner software, no logging utility, etc….you know the actually applications that were on the old hard drive and are required for making this whole thing work.

After about a 10 minute conversation with me, and then with his people back at his company, he started to do the load on the new hard drive.  This was completed, and then all the necessary applications were installed.  After about 2 hours everything was completed, tested, and working again as expected.

When I show up at a customer’s site, I am the expert, I would never dream of just showing up, putting in a new hard drive and then just saying “OK…done here” and leaving.

I don’t know if this was some kind of miss-communication between this tech and his office, or if he really just thought he was replacing a hard drive and only replacing a hard drive, but in either case;

He is the Expert ???????

You have probably heard the tag line “What can brown do for you today?” over the past few years;  well, Brown, I would like you to go fine people who actually can think about software engineering, and not just writing code.

They have a very good application called Worldship.  This software is critical to large customers, and also logistics companies for sending, tracking, shipping, and billing anything going out or coming in.  They have released a major upgrade to the software that is required for new rules and regulations in the shipping world.  For the past 5 years this software has allowed UNC path’s for the client configuration, now suddenly with the new version UNC path’s are no longer allowed.

WTF????  The support people said, well we have been having problems supporting UNC paths, so the programmers removed it.  OK, I have no problems moving to a mapped drive for this.  But, in order to install the upgrade, I have to do a complete uninstall of the old software, and then install as if this is new.

Again, OK, I can backup the database, and then restore the DB once the install is done, but wait…..What about the 5 years of customization’s we have created, what about all the import/export jobs that we have created, and are critical to the business process’?  The answer from support, those will all have to be re-created, sorry……….

So, Brown, what are you currently doing for me?  You are causing me 3 times the amount of work for an upgrade, and you are costing my client 3 times as much money as this should have cost.

Where should I send the bill?

That being said, I an just curious if anyone else is going to be dropping by to take a look at this.  I am going to try to get this going again, and maybe even change the idea/purpose behind this, maybe to better document some of the projects I am working on, but time will tell.

I must not be that good, if I was, I would have known that one of our customers decided that it would be OK for them to attempt replacing a Cisco Pix Firewall that was at the other end of their VPN connections.  Of course, at 4AM, when the VPN would not come back up……I found out. 

<Aside>….I seem to use this forum for a lot of venting…..

I fully subscribe to the project management theory of the 6 P’s.  Proper Planning Prevents Piss Poor Performance.  As well as the corollary… Failure to Plan on your Part, does not constitute an emergency on my part.

This is not to say that I fully plan and document every little step of what has to happen.  But I do plan the big steps, and try to identify those details that will have to be addressed to prevent problems.  For this project,  my plan would be.

1.  Get the current config off the old device
2. Review the config as a text file
3. Put this config on the new device, and see what needs to be “fixed”
4. Lock the configuration on the production device
5. Plan the downtime, and who needs to be notified.
6. Lock the config on the new device
7. Put the new device in place.
8. Test connections.

Pretty basic, I know the details of the IOS and how VPN’s work and who/how they are initiated.  VPN’s to this device are initiated by the other end, all this device does is listen for them, and accept.

As it turns out, the person putting the config on the new device kind of skipped steps 1 and 2.  He started on step 3, with what turned out to be an out of date configuration.  1 hour later, the VPN is up and working, and I am heading to the office on 3 hours of sleep. 

Please, learn to plan, or just tell me what you want done, and let me get you the plan.

Once again, I promise to try and be better about keeping this up to day.

This morning a customer with 4 different sites called in saying that 1 site could not reach the servers at their office.  Now this customer has VPN connections between all the offices to provide just a little bit of security to the data moving across the Internet.  When they said they could not reach the server on the private IP, but could reach it on the Internet, we knew it was a VPN issue.

For some reason, another vendor setup this VPN setup, and we never really looked it over.  When I got on-site, I was able to connect to the router (thanks to the ID/Password being taped right to it), and with no surprise saw the VPN was “disconnected”.  Forcing the connection to come up fixed the problem in about 3 seconds.

Whenever I setup a VPN connection I always make sure of 2 things;  1. Make sure you enable Keep-Alive, 2. always allow for on-demand connection (when the client tries to connect to a device across the VPN, if the VPN is not up, then bring it up.

Simple little steps like this prevent those little problems that alway’s seem to get in the way.

With these requests in mind we started looking at our network from the outside in.  The first focus was making sure that our customers can get to hosted systems all the time.  We have 3 different ISP’s with lines coming into our data-center.  ISP-A 1.5MB T1 line.  ISP-B 3.0MB bonded T1 with Ethernet handoff, and ISP-C is a 50/25 MB Ethernet connection.  In order to make sure that all system are available all the time, we purchased a FatPipe Warp 50 device.  This allows us to have all 3 ISP lines come into the WARD device, and then on into our network.  The Warp utilizes SmartDNS to provide DNS resolution with very low TTL’s so that if the primary ISP line is down, the request will be re-directed to another line within 15 seconds.

The physical connections were very easy to make, just move the ISP connections from the old router into the FatPipe.  We put in place a new firewall between the FatPipe on the outside, and a Layer 3 switch on the inside.  It took a little bit of time to get all the rules correct on the firewall, and all the routes correct on the FatPipe, but this part of the project finished successfully.  If you are interested in a more detailed description of how this is all working, please contact me.

The second part of the equation is the hosting environment itself.  This is a basic VMWare setup with an iSCSI SAN.    This process has not been easy, and I will detail it more in the next post.

I work for a Managed Service Provider, we have well over 100 customers, and have jumped headlong into hosting services for these existing customers, and as a hook to get new customers.  This have involved many new network designs, server setups, server moves, and a massive amount of problems due to a miss-behaving Storage Area Network device.  But this is a post for a later date.

Life, it is spring time here in the mid-atlantic area.  This means that soccer and yard work have both started up.  Currently I am handling both US Soccer club and tournaments as well as high-school soccer.  I also have 10 cubic yards of mulch sitting in my driveway. 

I promise to try to get at least 1 post a week, just don’t hold me to it.