Sidetracked While Heading to Left Field

While I will be the first to admit it has been about 14 years since I last touched an RS/6000, and in that time, the hardware has changed greatly, I do retain enough AIX and RS/6000 knowledge to know…..”The system won’t boot if you never installed the Operating System”.  It was bad enough, that it took 90 minutes to find a suitable USB keyboard, but then to finally get the console set and then watch as the boot sequence starting trying to find a network server, I then decided I had to laugh about this…..If I can’t laugh about it,I would end up in some perch shooting people.

The server was ordered through a second party.  They were responsible for the final details of the server config, and getting it shipped to my customer.  The first server actually shipped and was received, somewhere else, it was even signed for, but never actually showed up here.  So this second server was sent.  Apparently no one thought to recheck the list before putting it on a FedEx truck.  So…It must be time to do an install of AIX.  Damn voices in my head are laughing about this…..Partition this and setting that, it has been 14 years since I did this.

No problem….No Operating System is the box.  Well, that makes sense, if you don’t load it at the factory, why would you ship it with the server…On the other hand, the default ID/Password for IBM’s HMC is admin/admin.  Oh yeah, you have to change the password before the HMC will unlock all the functions.  For those who may care, or Google this…TO connect to the HMC on a new server

1.  Set you computers IP Address to 169.254.2.14 with a netmask of 255.255.255.0

2.  Plug an ethernet cable into the HMC1 port on the back of the server, and your PC.

3.  Point your web browser to httpS://169.254.2.147  (Note the HTTPS)

4.  Login as admin/admin

5.  Change your password.  You can change it back to admin

6.  Once this is done, all functions are unlocked.

Also, the RS/6000 firmware is not that big.  They don’t support USB fully.  This means you need a USB keyboard that is very basic.  Any extra buttons, and chances are the server won’t recognize it.

In the meantime, I will have to wait for the AIX install media to get here.  If you have any shortcuts or hint/tips for installing, PLEASE let me know.

Now Heading Towards Left Field

Who knows what a RS/6000 is?  I am now starting a new project at work to implement a shiny brand new IBM RS/6000 server.  It has been 14+ years since I have activly support this particular platform, and all I can think is “what the heck am I getting into”?  The system in question supports a medical billing platform.  As luck would have it I used to support this very software way back when it ran on Open VMS and even Vax/VMS.  

Some customers present challenges due to limited budget or limited grey matter.  This particular customer’s challenge is using technology that is all over the map.  Today I am setting up an RS/6000 and tomorrow I am migrating them to MS-Exchange 2007.  This should be an interesting couple of days.

Need to get back to this.

I really need to get back to this blog.  What can I say, work is actually getting in the way, and of course, work was the main reason for starting this.  So, lets talk engineering.

I work for a Managed Service Provider.  The main portion of the business is providing IT support and services for small to medium sized business.  We cover a rather large geographic area, and an even wider array of technology.  One of the area’s that we are rapidly expanding into is that of hosted services.  These are both customer applications running on our systems, as well as bringing customer systems into our facility.   In either case, access via the Internet is crucial to our success. 

We currently have 3 different Internet links that all come into their own router/firewall setups.  While this provides some redundancy, it really doesn’t help a company if they are only able to access the site from one ISP.  WE have undertaken a large re-engineering project to move all of these links onto a Layer 3 switch to allow for internal redundancy, and we are getting ready to move all ISP links to a Fatpipe connection.

The Fatpipe device is interesting in that it will handle the DNS needs (replacing our existing DNS infrastructure), as well as handling the redundancy across ISP links.    However, this does pose a new challenge.  Our outbound connections are expecting to leave the internal network on the same gateway they arrived.  If this does not happen firewalls will assume a spoofed IP Packet and drop it.  This is one challenge that we will need to address before putting in the Fatpipe.

Redundancy across ISP’s

One of the first projects I was given when starting this new job was to provide Internet redundancy to a customer.  They already had Internet access via ISP #1 (names change to protect the idiots).  They contacted ISP #2 and requested high-speed access and a block of public IP address.  While they were doing this, an Adtran NetVanta router appeared with 2 Internet connections and 8 LAN ports.  The idea was simple, replace the Cisco device at the company with this Adtran router, connect both ISP’s into the new router and provide redundancy.  This is how the project was presented to me.

Since I was new, and did not know much about Adtran capabilities, I said sure, and started down a long and frustrating path.  Problems with monitoring the active connection, the inability to set the “Active connection” was just the start of it.  As I moved past those issues, new issue cropped up, DNS round-robin was a foreign concept to the DNS hosting company, then there were Point to Point VPN connections that had to be created for both connections.  P2P VPN was an issue as the other end of the VPN would not accept multiple Peers to initiate the tunnel.  Then there were 4 client to site VPN connections to worry about.  

Engineering around the P2P tunnels was straightforward..  A simple ACL on the outbound ISP #2 connection would NAT the traffic so it looked like it was coming from another IP address.   VPN tunnels working.  Then the Client VPN issue was ready to be tackled.  All I had to do was un-install the Cisco software and install the VPN client from Adtran………….

3 weeks later, the VPN software does not work with Trend Micro Anti-Virus,   time to look at the Cisco hardware already in place.  ASA 5505 with Security Plus license, bingo…

3 months of dealing with problem after problem with the Adtran hardware, 3 hours and redundancy is set up on the Cisco device.  If ISP #1 goes down, ISP #2 takes over, since the 2 connections share the same NAT pool, it takes less than 2 seconds to fail-over, and the clients don’t even realize they have switched connections. 

All that is left is to have the 2nd peer address added at the other end of the P2P tunnels and this will be complete.

Long Time, No Update

I know, I know……It has been quitre a while since this has been updated.  I am not dead, yet .  I do however have a long fight with an Adtran NetVanta router that will be posted here.

On-Site….

The vast majority of my work is done at the customer site.  It has been an interesting twist from being the person who brought the “help” in, to becoming the help.  I am currently at my “problem child” customer, and wondering how this entire situation could have been avoided. 

Now, understand, this situation was created before I was even hired into this job, and it was just dumped into my lap.  Basically the customer wants Internet redundancy.  Sounds simple enough to them, and we agreed to make this happen.  the problem is that no one on either side of the discussion sat down to actually compile a list of what “redundancy” means.  So today’s class is on what “redundancy” actually means.

To me, being in networking for the past 18 years, I start the redundancy discussion on the far end of the scale.  This covers all aspects of making sure that a companies web presence (SMTP, HTTP, FTP, DNS, inbound and outbound) are covered.  A sample list is

• Who hosts DNS, will they add secondary and Alias records into your zones.
• Who hosts the MX records and how does mail flow into the organization.
• What incoming traffic is there to account for (HTTP, HTTPS, FTP, ETC)
• Do you want outbound traffic to automatically get re-directed if 1 interface goes down.
• How much are you willing to spend
• What problems have you had in the past year with the current provider

So, here I am, without any questions like this getting asked, trying to setup a brand new Router with multiple interfaces to multiple ISP’s.  My first problem is that there are existing VPN connections that were not identified (2 of them), so now I need to re-create 2 existing interfaces, and add 2 more VPN tunnels for redundancy.    The device I am working with (new router) is not  from a 5 letter company located in Cali…..I have been on the phone with this router’s tech support a lot translating Cisco speak into their speak. 

After about 15 hours of working on getting this configuration setup, I am now mere moments away from trying 1 of the new VPN tunnels.   As Lou Holtz would say on College Gameday “Not so fast my friend”  seems that the other end of the VPN tunnel is not as ready as he said he would be…..Need to reschedule for later in the day…….

ARGH…………………………………..

New Customer/Bad Son

Today, I actually brought a new customer into the store (so to speak). He was actually referred by another customer I had done some work for a few weeks ago. His complaint sounded simple enough, I have a PC that won’t boot. All I get is “Disk Read Failure, Press Ctrl+Alt+Del to Continue”. Sounded like a job for Frank the hardware guy in my head. Haven’t needed to deal with Frank these past couple of weeks, as most of my work has been networking, or simple setup/migrations.

Frank came out of the box on fire. Lets see what is not plugged in, who cracked this box and didn’t mention it, where did I put that damn screwdriver. Calm down Frank…… The system in question was an 18 month old Dell Inspiron desktop, a quick check showed that the warranty was still valid. Sorry Frank, put the screwdriver away.

A quick call to Dell and we start the diagnostic process, of course, every quick test passes. I thnk move on to the time consuming tests. Well about 90 minutes later, all comprehensive tests have passed, and the “owner” mentions that just before the problem started, they removed PGP WholeDisk Encryption.

Uh OH……………………………………………….

Quick question about how PGP was removed, and not a good answer, they just clicked what the “Owners” Son told them to click and then 24 hours later, the system won’t boot. Sure enough, I had a bare metal drive with a fully loaded MBR. A 30 minute restore of Vista Home they now have…..

A working computer

A good lesson in the importance of backing up more than once a year

The knowledge that not everyone who works with computers can fix computer.

And a new Service Provider. I think the $400 bill was going to be taken out of his sons allowance.

ISV Issues

One of the on-going challenges in the MSP space that I have to deal with is that our customers are squarely in the SMB space.  This means that for the most part IT budgets are very tight.  They know that they are already paying by the hour when I show up, so a lot of times this means that the other dollars available are reduced by my rate.
This means that I do a lot of research and use a lot of the applications and utilities that are provided by small Independent Software Vendors.  While these ISV’s are generally very good, and produce some excellent software, it also means that I am not able to get on the spot phone support.
Again, I am billing by the hour, so if I get to a roadblock and the only option is to send an email to the ISV, then I am guaranteeing the customer another hour of billing for when I have to come back to finally resolve the problem.
What are your experiences with ISV shops, and for those of you that outsource your IT, how does if feel knowing that you are going to get that extra time charge?  2 different arguments in my head over this;

Billable time is important to my boss

I am not able to fix the problem in 1 visit, and sometimes we won’t bill the second visit.

Seems like just another headache coming on.

Client Sites

Part of working with a Managed Service Provider is actually getting out of the office and visiting the clients at their site.  Clients to a MSP can come in all shapes and sizes.  So far I have been to a “corporate” site, the cubicle jungle, the quiet that is very loud, the kind of place most IT professionals are accustomed to.  I have also been to a client that you would never think of in terms of technology.  A little double-wide trailer way out in the country.

The challenge in the MSP space is that both of these customers are the same.  Data is data, and to the customer their data is critical.  I personally feel that I am technology agnostic, my wife may not agree, but I don’t care if you are using a Win-Tel platform or an Apple platform.  Windows, Linux, or Mac/OS, it doesn’t matter to me.

The MSP role in the world of small and medium sized business is truly that of PROVIDER.  I have the expertise that these customers need right now.  Once I am done, then they don’t need the expertise any more, and I am off to the next site.

This weeks voice has no name yet, but it is that nagging afterthought of……”Did you provide enough documentation to both the customer, and the next tech that may have to make a call”.  Lets face it, documenting the work done is tedious, boring, and always thought of last, but while you may not like doing it.  It may very well be the most important skill for a tech working for a Managed Service Provider.

Another Turn Around

On Monday the 13th, I started a new job with a managed service provider in my area. They provided any and all IT related services and support to small and mid sized business in the Mid-Atlantic area. I have come on board as a senior network/systems tech and am very excited about it. Not only is the position better than my last one, but I am back in an environment where I get to deal with everything from pulling cable all the way to high end networks, servers, and leading edge technology. This will also start exposing me to some different security work and tools. This week alone I have had to deal with your run of the mill trojans/virus, an emergency restore from backup for 1 companies AP/AR system, the joy of having to scrub up to deal with issues in a hospital pharmacy, as well as why a VPN tunnel was up and running, but the data was not getting to the tunnel. The VPN issue was due to a Barracuda Web filter. The voices in my head were yelling at me when I finally figured it out, that I should have know 2 hours sooner, and the dispatch at the new company was impressed it only took 3 hours to figure it out.