First networking related post

This is what happens when you have time to manage some cables:

BEFORE....

And this is what it looks like now (with two more 48 port patch panels)

There's been a year lapse between the two pics with the first being taken right after I took over the Network Engineer's position. Since then we completed the network core upgrade (moved from RIP to OSPF and implemented a redundant campus architecture) added a VNX5300, six ESXi servers (with 10 NICs each) and a partridge in a pear tree. I'm now going to work on removing the run of cables on the left and moving them down to a more modular patch panel design like on the bottom right. I can't wait to do this to the rest of our network closets.

Weird storage vMotion issue

I had a server template that I created on a ESXi 4.1 cluster and moved to a ESXi 5 cluster. After I did that, I upgraded the hardware version and remade it into a template and left it at that. I had been successfully deploying VM's from this template for a few months so I never thought there was something wrong. Well today I've been moving VM's to a new storage cluster on my vSphere 5 data-center using storage vMotion and when I got to this particular server, I converted it back into virtual machine and tried to vMotion it to the cluster. Well at 99% it fails with an "invalid configuration for device 12" error.

Huh?

Some searching turned up a buried post saying that it was an issue with the MAC address on the vnic. So I removed the old nic and added a new one and presto! Storage vMotion worked.

Weird.

Comments...

Apparently they're broken. I need to fix them.

edit: fixed them.

Off in the Horizon...

Application virtualization has been on the rise for the last few years with all the different mobile platforms available. Citrix has been doing it for years... well they've been presenting users with a remote app type environment at least. You can't really call that "virtualized" when the user has an entire session on a terminal server all to themselves. VMware's Thinapp is a whole lot different though. When you take an entire application and box it up into an MSI package then stream it to a user... that's what I call virtualization. So the question is how do you present that Thinapp to the end user? VMware's solution is their Horizon Application Manager.

I've been a little leery of actually using Thinapps in our environment. One reason is the fact that it is a shared application package and I don't know how it will react to 300 users accessing it at the same time... but you can't find solutions to problems by sitting around worrying about the outcome so I'm delving into it a little at a time.

Last week I decided to give Horizon Application Manager a shot and see what it was all about. I needed a solution to start streaming some applications to virtual desktops so that I wouldn't have to install the software on each of them (which can be a pain). I read through the documentation and some blogs about installing it and here are my reactions to the installation process. Please note that these were jotted down as I was installing the product and therefore are in chronological order. The things I had issues with in the beginning I ended up fixing later on but you'll see what I mean.

- When deploying the template, you need to assign an IP address to the virtual machine but when powering on the machine, there must be a pool of reserved IP address on the virtual switch of the VM or else the virtual machine won't power on. But the IP address that is assigned to the VM can't be a part of that pool even though it won't be assigned an address within that pool...

- When setting up a test environment and you do NOT use the connector virtual machine (which is allowed) the local user account that gets created doesn't seem to be allowed to log in. If a password gets emailed to the administrator email, there is no option to set a SMTP server to sent the email through... and if the virtual machine assumes that it can just use the outside MX records, that's a pretty big flaw. The end result was me not being able to log in and having to delete the VM and redeploy from the template

- when setting up an organization and you wish to change the organization's name, you can't go back in the set up. you're stuck permanently with the name you've chosen

- when setting up the conntector it won't join my domain giving me a "Failed to join the domain: ERROR_INVALID_COMPUTERNAME" error.

- apparently I missed the part where during the connector set up they said it was vitally important to active directory sync to set a hostname. No option to go back and change the hostname anywhere on the connector console or in the web ui.

- when setting up the system to join our domain, I had to log into the SLES console, change the hostname and edit the /etc/HOSTNAME file. Only after I changed those values and rebooted was I able to join the domain. not fun.

- After enabling windows authentication (after I've already completed the wizard once without activating it), access to horizon stopped with a 404 error. can't log into manager. I may have to delete the connector in App Manager...

- No warning about loosing connectivity to app manager and needing to put the activation code back in after deleting the connector... and now I have to figure out how to regenerate an activation code from the app manager console

- Found out that there is a built in account named "operator" whose password I set earlier. Where was that in the documentation? 5+ hours wasted.

- No automatic agent installation from the browser... I had to go get it from VMware's site.

- No way to remove a Thinapp from the list of available apps without going into the console and deleting a file. That's kind of crazy

- The application management itself could be way better.

So the end result was finally a working Horizon installation. Repackaging previously packaged apps to work with Horizon is a little bit of a pain but it worked.

Long time no post

I apologize for the absence, but work/studying and wedding planning has taken all my time recently (not to mention I screwed up the template on this blog and after that I just didn't really feel like fixing it). I did end up taking an passing the VCP5-DT exam last month so I'm now officially a VCP5-DT. I've got a few things to write about here soon like my VMware Horizion Manager installation experience and studying for the CCNP ROUTE exam. I'll post more later.

So, what's next?

VCP5?

Done! :)

Guess I should work on VCP5-DT since it should be short and sweet (been using View since last year and I've gotten pretty intimate with it)

View 5.1 BUG Alert!

I found one!

I just upgraded our environment to the latest build of 5.1 over the weekend and in the processes upgraded the agent to 5.1 on our base virtual machines. On Tuesday, one of our users (who's using a zero client) reported they could no longer change the resolution to their desktop. I checked all the settings as recommended by KB article 1020809 and even though I could move the slider on the display settings (it is a Windows XP virtual machine) and then apply it, the display would then go dark like it was changing the resolution and when it came back, the resolution would still be the native resolution of the monitor. I even tried a little documented command:

VMwareResolutionSet.exe 0 1 , 0 0 1280 800

(btw, that command is located in "c:\Program Files\Common Files\VMware\Teradici PCoIP Server\")

which sets the resolution on the first monitor to 1280x800... no luck. Same outcome as before. One of my co-workers who also has a zero client and a Windows 7 desktop tried to change his to test and his wouldn't change either. The only resolution we could change it to was 800x600.

When I dropped the user down from agent version 5.1 to 5.0, they could again change the resolution settings. I told VMware about the bug but so far, I haven't received any type of response back from them. Hopefully they'll have a fix out for it soon.

update: Turns out it was the firmware version on the zero client. As of the time of this writing, there wasn't a firmware that supported View 5.1. This issue has since been fixed.

ESXi, vCenter and Active Directory

I don't have any confirmation on this (in fact VMware support has no idea how this happened), but after spending 11 hours on Saturday rebuilding one of my vSphere servers (which houses the entire View environment), I think I figured out why 90% of my vm's went orphaned on me after I took the server offline for some hardware upgrades (like ironically, a fibre HBA to hook into our new EMC VNX5300).

When I originally built this vSphere environment, I did so with the thought that this would be all I get to work with. I didn't have any shared storage (not even a decent iSCSI/NFS store), nor did I have vMotion... and apparently I didn't really know exactly what I was doing ("gasp!" - getting that out was hard for me because I hate to think that I did anything wrong). I had an idea of what I wanted but VMware was still a little new to me. Fast forward six months, and I've gone through "Mastering vSphere 5" twice, read most of VMware's vSphere 5 documents and have taken the vSphere 5 class (my VCP test is registered for June 4th). I have a MUCH better understanding of how the whole vSphere environment works. So this is how I think I killed 96 virtual machines - causing me to re-register each and every one of them and taking down my entire View environment.

I added the ESXi host to vCenter using a domain account whose password changes every 60 days.

See, I went through and set up access to the ESXi host before adding it to vCenter. I really didn't understand how vCenter played such an important role in management until recently. Here's my train of thought (granted, none of this is verified - just my own suspicions):

• I add the ESXi host to vCenter with a domain account
• vCenter cache's the username and password used to connect and authenticate the ESXi host
• I add some VM's to the host through vCenter
• vCenter uses those cached credentials to register those VM's with the ESXi host itself
• My domain account's password is changed
• I then proceed to build out my entire view environment on this host
• Each time a VM is created in vCenter, vCenter adds it to it's database then tries to register it with the ESXi host using the original cached credentials
• uh-oh... those credentials are expired. vCenter gets to actually create the virtual machine files on the datastore but the ESXi host refuses to add the VM to its list of hosted virtual machines
• Fast forward to Saturday and I take the host down to put the new HBA cards in it
• When I bring the host back online, vCenter tries to re-add the host but is unable to do so using the cached credentials
• I have to re-add the host to vCenter, this time realizing I need to use the ESXi host root username and password.

Again, this is my speculation, but it seems to be the only logical answer to how this happened. I mean, there's no way taking a host offline on purpose should make 90+ VM's orphaned you know?

Anyways, if anyone ever reads this and they know of official documentation from VMware that supports this theory, I'd love to read it!

Hostnames and View Volatile Environment Registry Entries

We're starting to deploy a large amount of linked clones that require the zero client machine name to base network printer mappings too. I created this batch script to grab the volatile environment machine name key and shove it into a environmental variable but before I go on with the story, here's the batch script if anyone can use it for something else:

@echo off

FOR /F "tokens=4 delims= " %%A IN ('whoami.exe /user /sid') DO SET RegKey="HKU\%%A\Volatile Environment"

c:\windows\system32\reg.exe copy %RegKey% "HKCU\Volatile Environment" /f /s

c:\windows\system32\setx.exe (your variable here) -K "HKEY_CURRENT_USER\Volatile Environment\ViewClient_Machine_Name"

So a few weeks go by and the new guy on the staff get's asked to build another script to map the network printer... now he's never worked with VMware View before so keep that in mind. I go off to fix some end user issue and the next time I see him two hours later he's still working on trying to write the script but he asks me to come over and take a look at something so I do. He then proceeds to type in:

echo %ViewClient_Machine_Name%

into a command prompt and wouldn't you know... his zero client's name came up. I was like "wha??". So I had him log into his desktop using his iPad and try it again. Low and behold, the variable changed. I was like "no way". I had literally worked a half a day on that above script to get the variable created based on the key location within the HKEY_USER sub section of the registry. I guess my next question is why isn't that info documented more... The fact that all those volatile environmental entries in the registry are already made into environmental variables slipped past not only the View class I took but also 98% of all web pages out there. Grrr.

quick way to break a virtual desktop...

Just FYI. If you take ownership of the c:\windows\system32 directory away from "nt service\trustedinstaller", you'll break vmware view.

Quick Java commands to compile a jar file

This is almost for just me, but maybe someone else could use the info as well.

To compile the source code into class files (I didn't have the JDK directory in my path, hence the long command):

c:\<directory to jdk>\bin\javac.exe c:\<src directory>\file.java

Once you work out the bugs to the class file (errors and warnings) you create the jar file with the following command:

c:\<directory to jdk>\bin\jar.exe cvfm <name>.jar manifest.txt com/

The "cvfm" portions are flags for the jar compressor. <name>.jar is the output file name you want. Manifest.txt contains the directory listing of class files. com/ is the base directory of your class files.

Lessons Learned

User logs into their View virtual desktop using the EVGA PD02 Zero Client (awesome machine btw), spends a few hours doing something then disconnects from the virtual machine and goes to lunch. When that user gets back and tries to re-log into their disconnected session, the virtual desktop kicks them out and the screen goes right to the login prompt on the zero client again... W...T...H...

It literally took me a week (and unfortunately a call to VMware support) to figure it out. No other person that's been using View had this problem. I uninstalled and reinstalled the View agent. Deleted and recreated the pool (twice). Tried every PCoIP and View agent group policy setting I could find that might remotely have anything to do with disconnected sessions...

nothing...

called to VMware support went like this:

VMware Rep: I see your problem is *(removed for brevity)*. Let's do a Webex.
Me: Awesome - here you go
Rep: Ok, let's first check the power settings...
Me: *(sound of hand slapping forehead in a moment of clarity)*

turn the damn monitor sleep settings in windows 7 OFF or else you won't reconnect to the session using PCoIP. It seriously took me a week to get that problem fixed.

So sometimes calling support is the right thing to do... and they're not all idiots

Lesson learned.

SANs amore

Incoming EMC VNX5300 SAN



I'm pretty stoked about this. I'll post more once everything is finalized and talk about the design concepts I've been dreaming up plus how everything will be connected from the ground/network on up.

Stay tuned!

Next solution please...

I have a real disdain for level 1 tech support. I know those people work hard and that they more than likely don't get the right kind of training to support questions from people like me, but can I have a level 2 tech support pass for every company I deal with? Please!? I get irritated when I have to spend 45 minute fielding questions such as, "did you reboot the pc?"

ok... /rant off

That had to do with calling Imprivata support support earlier today. It wasn't as bad as getting asked if I rebooted the virtual machine, but the guy on the other end clearly was having a problem fully understanding what I was trying to convey to him.

I was at the end of building a Windows 7 virtual machine for a contractor of ours and for some odd reason when I would sign into the View connection server then log into this desktop, Imprivata would pop up their logon UI asking for me to put in my credentials again. Kinda annoying. So the good tech on the end of the line spent about 15 minutes trying to decipher what I was talking about so that he could put some keywords into their little KB article search engine and try to find an answer for me. Well, like always there was no KB article for my problem, so he took my info from me and told me he'd look into it. Two hours later I get an email from them stating that the solution to my problem was a registry hack to use Kerberos authentication only. "Okay... might as well try it..."

Nope, not it. So I went digging myself and found http://portals.imprivata.com/ftp/SFAttach/Credential_Provider_Wrapping.pdf

after checking out that ISXCredProvDiag tool on the virtual desktop, I saw that the VMware Agent wasn't hooked.


so I wrapped it up, rebooted and BAM... authentication passed through from the connection server. Sweet.

Imprivata OneSign Agent version: 4.5.54.54
VMware View Agent version: 5

View Annoyances

I love me some linked clones and Persona Management in View. Absolute heaven. I honestly can't think of a better way to deliver a virtual desktop. That's not to say it doesn't comes with some annoyances... especially if you just use the linked clones and NOT the persona management. For example, we have a health management program for our hospital that uses java over a web browser to deliver the end product to our nursing staff. One of the little annoyances that I had with building the virtual machine in the previous post was that every time I got a new desktop and started up the web app for our health management program, I would get a prompt from JRE (Java Runtime Environment) to accept the self signed certificate the company used to sign the java program. Well, after much digging around I figured out how to install the certificate into the java store and explicitly trust it forever - therefore not needing to accept the certificate each time... overall it saves a few mouse clicks for the end user. Here are the steps:

1) copy the self-signed certificate to the folder on the machine (example: c:\java_cert\signedcert.crt)

2) if you have a default 32bit JRE 6 installation, navigate to the "c:\program files\java\jre6\bin" folder in a command prompt. Modify as necessary for 64bit installations.

3) type the following command to import the self signed certificate (again, this is for a default installation):
keytool.exe -importcert -trustcacerts -keystore "c:\program files\java\jre6\lib\security\cacerts" -storepass changeit -noprompt -alias -file c:\java_cert\signedcert.crt

4) create a file in the "C:\Windows\Sun\java\deployment" folder named "deployment.config"

5) in that file put the following lines:

deployment.system.config=file\:C\:/Windows/Sun/Java/Deployment/deployment.properties
deployment.system.config.mandatory=true

*each slash is needed as the backslashes escape the colons that are next to them*

The first line points to the current deployment properties files and the second line makes it mandatory that it be used (helps with locking down java apps if needed). Setting it to "false" will tell JRE to try to use it as it launches a java app, but skip it and revert to defaults if it doesn't exist. You can read more about these files and their options here on Oracle's website: Java Deployment Guide

6) create a file in that same directory named deployment.properties (as specified in the deployment.config file) and put these lines in it:

deployment.system.security.trusted.certs=C:\\java_certs\\trusted.certs

*the double slashes are needed because the first slash escapes the second one each time.*

7) under your username, go ahead and accept and trust the cert for the java app you're trying to run.

8) do a search for "trusted.certs" under your user profile folder. (on windows 7 it's located at C:\Users\(your username here)\AppData\LocalLow\Sun\Java\Deployment\security\trusted.certs)

9) create a folder on the C: drive of the virtual machine and name it "java_certs" (the name doesn't matter, but it needs to be the same name as the folder in the deployment.properties file)

10) copy your trusted.certs file over to that new folder

Now that should be it. You should be able to run the java app in the browser for every user that logs into that machine from now on without them having to accept and trust the self signed certificate.

Do they actually need it?

We've been thinking about replacing some of our ancient computers on wheels (COWs) over at our hospital facility with wireless Wyse devices and View virtual desktops for a few months now. The current ones are ancient mobile P4 based machines that we can't really get parts for. They're set up as locked down Windows XP desktops with very minimal rights.

Since I can't really use XP any longer and our move to Windows 7 eminent, I've been working on a special Windows 7 virtual desktop for our auto logon PCs down at the hospital when I realized that, through group policy I had locked down the users so much that the only link on the start menu was to the printers...

So I asked one of my colleagues about the start button and menu... Do the users actually need it? I mean, they really only ever use the shortcuts on the desktop, and for auto logon/locked down desktops, there's really no need for it. So I set about the task to get rid of the Windows 7 Start Button/Ballon (whatever you want to call it).

I honestly searched high and low for a quick answer to the getting rid of it like through Group Policy or a dirty reg edit or what-have-you... but all I found were posts about making Windows UI API calls that hid the button - like they did with the original XP start button. My hope for doing this the easy way quickly died. Well, after attempting to editing a C# program that didn't work (I have no idea how to write C# but programming to me is pretty logical and given a little bit of time I can usually figure out what any piece of code is actually doing) that I found on one of the bigger code sharing sites with no luck, I found a simple little program called "StartKiller" from Tordex.com. Somehow, they were able to find a way to completely hide and remove both the start button and menu in windows 7... I was floored. Not only that, but it also hides the tray icon as well! I found out that just running the program at logon through group policy wasn't enough to hide the start menu and tray icon so using some procmon magic from sysinternals, I found the registry key they use on the fly to do both. I exported the key to a reg file and using the amazing program called RegToAdm (from the NUTS program package... awesome program. I haven't found a need for the other utilities yet, but RegToAdm is fantastic) to create this custom ADM file (please note - this ADM file isn't perfect and could be better):

----------------------------------------------------------------------------------------
CLASS USER

CATEGORY "Start Killer"
KEYNAME "Software\True Software\Start Killer"

POLICY "ShowTrayIcon"
EXPLAIN "This value is either 1 or 0. Set it to 1 to show the tray icon or 0 to hide it."
PART "ShowTrayIcon"
NUMERIC
VALUENAME "ShowTrayIcon"
MIN 0
MAX 1
DEFAULT 0
END PART
END POLICY

POLICY "BlockStart"
EXPLAIN "This value is either 1 or 0. Set it to 1 to disable the start menu or 0 to enable it"
PART "BlockStart"
NUMERIC
VALUENAME "BlockStart"
MIN 0
MAX 1
DEFAULT 1
END PART
END POLICY

END CATEGORY
----------------------------------------------------------------------------------------

So, setting start killer to run at logon and adding this ADM file to the user GP gave me this


Fantastic, no start menu, no worries... just find the icon on the desktop and click it (icons removed for privacy)

We're at a point with our VMware infrastructure (and the one we're about to build) that it was time to find a better anti-virus solution for the virtual machines. I am really anxious about running a full blown client in a VM with questions such as:

1) how many resources could it consume
2) how will we update the signatures on VMware View linked clone virtual desktops
3) did I mention the question about resources?

So I called up Trend to talk about their Deep Security product. I wanted to test it out to find out if it was going to be worth the trouble, as in moving away from CA (our current AV solution) or working in conjunction with it (eg. Trend gets our virtual environment and CA gets the physical). Well, we decided to try a test run on some dummy VMs and a host I could hot add to the production vSphere environment. We got everything set up (server 2008R2, a few virtual desktops, vShield Manager, VMware Tools 8.6 with the Endpoint driver) and started to install their version 7.5 (because 8.0 isn't compatible with vSphere 4.1u2 yet) and then they tell me that I can't have vShield Manager 5 installed on vSphere 4.1 because Deep Security 7.5 isn't compatible with vShield 5... really!? By the way... in a month and a half, Trend is coming out with a service pack for Deep Security 8 so that it's backwards compatible with vSphere 4.1 AND vShield 5...

So instead of dropping down to vShield v1 (which is the version for vSphere 4.1) and then re-upgrading to vShield 5, I just decided to build an entire new lab for just the Trend proof of concept.

What does two Core2Duo boxes with 8 GB of ram each and one Core2Duo box with 2 GB of RAM get you?

This:

1 server 2008R2 domain controller with DNS
2 vSphere 5 servers running
-1 vCenter Server
-1 Trend Security server
-2 Windows 7 virtual desktops
-1 vShield Manager
-1 Deep Security 8.0 virtual appliance

All built entirely in a day. We just got done with the install today and I'd have to say, I'm pretty impressed with Trend's capabilities. We'll see how it actually stands up to a thumb drive full of viruses and no connection to the internet tomorrow :P

Hello World

Cliche, I know...

But then again, doesn't everything in the tech world begin with "Hello World"?

I'd like to give the world a little introduction to me. I began my journey into the world of tech when I was very young with my dad's first ColecoVision ADAM computer system back in the early 80's. Ever since then I've been enamored with computers. I built my first one in the days of the original Pentium. My friends introduced me to DOOM when it first came out and since my dad had the fastest computer, we would spend hours tweaking his autoexec.bat and config.sys files trying to get the most juice out of that system as possible, just to play some video games. Finally my dad had enough of me messing with his system that he took me to a local computer show and with $200 we bought a Cyrix 5x86 P266 processor, a motherboard and a couple sticks of RAM - and we're talking about original SIMM's... of the 2 megabyte parity/non-parity type... oh yeah, old school baby! When it was all said and done, I had a 200MB hard drive, a pretty decent proc, 8MB of RAM and a CD-ROM. Best part was, it ended up being better than my dad's and it hasn't stopped since then... and I still have a better system than him.

I got into IT a few years back when I was working as a tech at CompUSA. I realized while I was there that being in the repair business wasn't going to get me where I wanted to be professionally. So I went back to school, got my associates in networking (Cisco style) and landed a pretty nice starter position here at my company as a tech, but I was also brought on to help with the Cisco networking.

Since I've started, things have happened pretty rapidly here (more so than I could have ever dreamed) that lead to them offering me the Network Engineer position that I'm in now. I've learned so much in my time here that I just can't keep all the information to myself any longer. When you're working in a fast paced and wholly understaffed IT department, you have to find off the wall solutions to many, many problems. I've also come up with a few myself. I've gotten the help from so many anonymous people on the internet, that I feel it was time that I started giving back. Joining one forum just wasn't enough, and I don't have time to be a part of ten, so I figured if i just started this blog and handed out my solutions to issues I've come across then maybe, just maybe, I could give back to the internet community that had given so much to me in the past.

So here I am... at The Network Core. I hope you find what you're looking for.