Fridays in IT: Read-Only Friday

Read-Only Friday.  It’s an IT tradition that I really think every company should abide by.  Seriously, I really think every IT company should have it in a contract or something.  What is it?  Well, Friday is the last day of the week…so you should do everything in your power to not have a terrible weekend by updating or making any changes to your production environment on a Friday.  Friday in the IT world should be for emergency calls and finishing up projects you had during the week…not for installing that new software for your users or installing that shiny new UPS to replace one that is working, just old.

Why the fuck is this important?

Well, if you install new software (or god forbid, hardware) on Friday you have to make sure that it fucking works by the end of the day for the user.  Now, using new software might also be easy for you, so after installing and testing it you’re done, right? NOPE! Remember, the user is used to the old stuff, so they’re not going to know a damn thing about the new software you just installed for them most likely.  They’re going to call you for every little thing they need help with..EVERYTHING.  And if they can’t get something working, you’re stuck helping them.  Now, if the program install doesn’t go smoothly…you’re screwed.  You’ve gotta make sure it’s working for Monday, so guess what?  You’re staying until it’s working.

It’s worse for hardware.  If that brand new thing doesn’t work right (and you know it won’t) you’re stuck there..and if you think you’re good on Friday..wait.  You’ll get that call on Saturday that you NEED TO BE AT WORK TO FIX THAT NEW THING CAUSE IT’S NOT WORKING RIGHT!  Yeah.  It happens more often than people think.

So..Friday is really the worst time to do ANY upgrades or installs in IT.  Seriously..I just was told by a client who isn’t here (sick) to install a new version of Adobe Creative Suite on a users’ computer.  CS6 changed a whole lot from CS5.  So, like a moron, I do it.  I’ve now been called 3 times already by this person because they don’t know the new layout of CS6.  I’ve had to go across campus 2 times for this person.  I’m waiting for the phone to ring again…I just know it’s going to happen.

Why did I install new shit today?!?

Well, being client work, I have to make the client happy.  If that’s what they want, that’s what they get.  One good thing though?  If something else goes wrong after 3PM (when I’m leaving) they have to deal with it themselves. :-)

So please, everyone.  If you work with IT people, don’t ask them to do major shit to your production environment on Fridays.  It’s a terrible fucking idea for everyone involved, especially the IT staff.  I’m not even joking when I say that if I ever have my own IT staff, I’m putting read-only Friday as a real thing, mark my words.  Friday is for planning, emergencies, and finalizing work. That’s it. Nothing else.

Now I’m going to get ready to go home and have a beer.

A New Job, a New Life

My last update back in February was going to have a follow-up post, but I got a little side-tracked with my life.  Without going into much detail, I had a big change in my life for the better and I needed to move away from New York in order to complete my change and leave all of my negativity behind.  In February I decided to move back to Philadelphia and with this move I would have to change jobs of course.  I was looking for work for a few months before I moved (I knew I was going to move back many months before I actually did I just needed the job to come in before I could finally do it), and I received several interviews from very reputable companies and institutions.  I was very close to working for another college, but due to circumstances that were beyond my control that job fell through.

I few weeks after that I applied for a job working for a technology consulting company that d0es work for independent k-12 schools in the Philadelphia Metro-area.  I received a call-back the very next day and set up and interview within the next week.  Three weeks later and I had a new job waiting for me on February 27th as a Senior Support Engineer for this company (I will leave my company name private for the time being).

My new job is similar to my work at Poly but also very different.  I’m working with multiple clients, visiting different schools throughout the week and basically solving their tech issues both big and small.  It’s a lot slower-paced than I’m used to, and the levels of work can range from basic software updates and printer repair all the way up to large scale wireless and network deployment and server room build-outs.

I have clients of my own (only about 2 right now) and in 6-months time I will be managing several more locations and possibly managing more of the company as a whole (scheduling, getting new clients, purchasing for schools, etc.).  Its a very small company, just 4 full-time guys and one part-time, but there’s usually always something to do or someplace to be.

I don’t want this post to be ALL about the new job, so I’ll deviate a bit from that and mention that I did buy a car.  It’s the newest car I’ve ever owned, and I love it.  I did have to buy it because of work, but I wanted my own vehicle for some time.  Anyway, it’s a 2005 Mazda 3i Sport, it’s metallic Grey with an interesting blue pinstripe.

Adzam the Mazda

My new ride!

I love driving, and this car is fun to drive.  It’s one of the smaller cars I’ve owned but I don’t need to carry 5 people around anymore..all of my friends drive now!  Hopefully I’ll be able to put some customizations in it; maybe an in-dash navi/screen, but for now, it’s prefect for me.

Anyway, I will try to update more with my new job fun.  I’ll probably have to be vague about some details of work (you know… k-12 schools and all) but hopefully I can sneak some writing time in more often and maybe some different types of updates (maybe car mods/work!).

Upgrading from 10.4 Server to 10.5 Server

As I’ve mentioned before, I manage 2 Apple Xservers and a lab full of Macs.  When classes ended for the summer I decided to upgrade our Tiger server to Leopard.  This was a task that I knew was going to be a lot of work especially because I was consolidating our old Dell web/mail/DNS into the Leopard machine.  These computers were set up well before I came into the picture, and I was still learning new things about them as time went on.  For example: I didn’t know that the old system (Wanda) ran another DNS on top of the university DNS.  Confusing.

So, I decided to do this some time ago when I realized that every time the server would restart passwords would be locked out.  This was an issue with the Kerberos setup in Tiger which was half broken by the time I got here.  Long story short, I wanted to start over, so I did.

Upgrading from one system to another on a server s never an easy task, and Leopard was definitely no exception at all.  I’ve upgraded Linux systems much easier than this.  I decided to make the image in the lab one of the PowerPC G5s we have (the server is a G5).  Here’s a fairly quick rundown of how I got everything to run like I wanted to.

  1. Install 10.5 Server fresh selecting “Advanced Setup” from the choices offered.  We’re going to be hosting a ton of services on here and I want to be able to configure every single one of them myself.
  2. Install ALL updates available BEFORE SETTING ANYTHING UP!  This is something I always do.  it’s so much easier to start setting up the server after it’s completely updated.  I don’t want to set up then have something break because the new update fucked it up and believe me, it happens.  During this time I also updated XCode to 3.1.x so I can build anything with MacPorts when needed.
  3. Next I needed to re-install or re-compile PHP5 so I had GD and MCRYPT.  Instead of recompiling it I decided to install the Entropy PHP build which comes with all that jazz installed plus a few more things.  All I have to do is disable the apple libphp5 plugin that comes wth Apache2.  Done.
  4. Next I installed the Autodesk Maya FlexLM license server.  This was a bit annoying since there technically is no Leopard build from Autodesk.  I decided to download the License server from Autodesk and go into the installer package and snoop around a bit.  I found a conf file that I could edit to allow it to avoid checking for 10.5.  That allowed me to install part of it, but the license server wouldn’t go.  I finally just went into the package and took the launch daemon out and the library and just copied the files to their respective places in the system.  Rebooted and it worked fine.
  5. I decided to setup phpmyadmin as well as Webmin for administration.  Webmin is great, I’m very used to it and it’s very lightweight. Phpmyadmin is obvious.  I copied the folders to /etc and redirected the links in Apache using server admin and ran the setup scripts. I also installed the OS X Web-based open directory password changer while I was at it.  All worked fine.
  6. Next I did an rsync from the old webserver for the website.  Copied everything over, dumped and restored the MySQL databases and users and voila! Web is back.
  7. I rsync’ed the Mailman mailing lists from the old server.  This was another issue.  The data, archive, and list confs were all compatible; however, the symlinks on the copied lists were not working because they were still pointing to the old server model.  I couldn’t really change the links without mailman complaining so I did some workin around this.  First, I copied the archives from the old server to the new, this worked fine.  I then exported the lists using server admin export AFTER I copied the lists from the old server.  This gave me all mailing lists but nothing else, which is what I needed.  I removed all the old lists and confs from the system but left the archives in place.  I re-imported the lists that I exported from server admin and BING!  All mailing lists recreated the links and all lists were back to their working state.  Archives came over, user lists came over, and no complaining mailman.
  8. After that was set I copied all of the DNS settings from the old server and put them in the new DNS.  I also gave an updated list to Poly’s IS and disabled the now deprecated IDMI nameserver.  Everything was now BXMC.  This was about the time that I finally turned off Wanda, the old webserver, for good.  Wanda will be reconfigured for Hans.
  9. I enabled everything after I turned off Wanda; DNS, Firewall, DHCP, Web, Mail, etc. DHCP and Firewall were imported over from the Tiger install, so that was nothing.
  10. I attempted to restore the open directory database from the Tiger backup.  It wasn’t working, at all.  I was getting frustrated because now I was thinking I had to put every user in by hand.  I finally figured out that I was attempting to restore the LDAP database using the new kerberos realm and new LDAP searchbase.  This won’t work because the backup is using the old one, they just won’t talk.  I tried to restore using our old searchbase and realm and it worked!  I did attempt to change the realm and searchbase with no success, so for now I’m leaving it at IDMI.  No one will see this except myself and the systems so I could not care.
  11. With OD working again I tested all logging into the wiki, website, lab machines, afp, etc.  All worked after a password update (which all users will need to do unfrotunately).
  12. The final bit was to get the NetRestore working on the server.  I wrote about using NetRestore by Mike Bombich here, and how it reached EOL.  I stated that I would continue to use it until it stopped working….well, it stopped.  I was sad to see this actually, but our mac’s wouldn’t boot from the system anymore.  They all failed for some unexplained reason.  I decided to give the one Mike Bombich recommended a shot, DeployStudio.  Installing DS was easy, just a simple pkg file.  Setup was even fairly easy.  I ran the server setup on the server and that’s really it.  I used carbon copy cloner on a labmac to create an image (even though DS allows you to create one in the NetBoot screen, it didn’t work) and the NetRestore was back working.  I actually would like the wrote more about DS in the future.  I am very surprised with how well it works and the features it has built in. But that’ll be another story.

So this entire process took me 4 fresh installs on a lab machine before I got the image the way I wanted it (about 4-5 days).  It took 45 minutes to copy the image from the external hard drive to the server, and took about 3 days of tweaks to get everything running smoothly.  The entire experience was pretty painless, but it kept me busy for the entire time.  I will be re-imaging the secondary server again (it was upgraded to Leopard before the main server last summer) so that I will now have everything fresh and clean for the next semester.  Here’s a nice shot of server admin with everything running nicely:-)

picture-3

If anyone reads this and needs help with a server setup and/or upgrade from Tiger to Leopard please feel free to comment or ask me any questions about it because I’ve seen every issue now with the server and I now know how to get around a lot of things with it.