I moved!

Come visit me @ http://www.the-network-core.com!

Link Aggregation on a 3Com 4200G

As I've been building up my lab, I needed to get a backbone switch that would not only support my production/lab network traffic but also iSCSI and to do iSCSI right you need jumbo frames. Lo and behold, I was able to get my hands on a 24 port 3Com 4200G. You might wonder why I would want such a thing (and trust me... it's not your father's Cisco) but I was able to get it for free and it's fully managed (and they're only about $150 on Ebay). HP actually recently released an updated firmware and boot code for it which fixed a few bugs (get it here)- and it was their first one in something like five years. These might be made by the black sheep of the networking world (I don't know many that actually like 3Com switches) but they support many of the functions that I'll need for my lab - namely link aggregation and jumbo frames.

Speaking of link aggregation, let me tell you about how I thought 3Com implemented ether-channel and then actually show you the hidden way to do it. Basically my set-up is as follows: I have an ESXi host with four NICs. Two for production traffic and eventually the other two will be for iSCSI. I'm running 5.1 and my vSwitch is set up as follows:

Pretty standard - nothing extra going on. So I head on over to my 4200G and create, what I thought was, a load sharing link-aggregation group.

[4200G] link-aggregation group 1 mode manual
[4200G] interface GigabitEthernet 1/0/11
[4200G-GigabitEthernet1/0/11] link-aggregation group 1
[4200G-GigabitEthernet1/0/11] interface GigabitEthernet 1/0/12
[4200G-GigabitEthernet1/0/12] link-aggregation group 1
[4200G-GigabitEthernet1/0/12]

Now since ESXi doesn't support LACP without a vDS and I wouldn't want to use a vDS on my management network, etherchannel is my only option and I thought what I was doing created that. Well turns out I was wrong. I went to deploy my vCenter appliance and then went over to see what my networking usage was out of the blue. Suddenly I noticed that my second NIC... hadn't been used at all over the last hour and I know it should've been. I started to investigate between the 3Com's config and the setting on the ESXi server. All seemed correctly set up but just the odd behavior of all the traffic going down one pipe for every VM that I created. I did a "show interface" on the 3Com port that my server's second NIC was plugged into... I then saw something interesting:

[4200G]display interface GigabitEthernet 1/0/12
 GigabitEthernet1/0/12 current state : UP
 IP Sending Frames' Format is PKTFMT_ETHNT_2, Hardware address is 0024-731c-1d4e
 Media type is twisted pair, loopback not set
 Port hardware type is 1000_BASE_T
 1000Mbps-speed mode, full-duplex mode
 Link speed type is autonegotiation, link duplex type is autonegotiation
 Flow-control is not enabled
 The Maximum Frame Length is 9216
 Broadcast MAX-pps: 1488100
 Unicast MAX-ratio: 100%
 Multicast MAX-ratio: 100%
 Unknown Multicast Packet drop: Disable
 Unknown Unicast Packet drop: Disable
 Allow jumbo frame to pass
 PVID: 1
 Mdi type: auto
 Port link-type: trunk
  VLAN passing  : 1(default vlan), 2, 4, 20
  VLAN permitted: 1(default vlan), 2, 4, 20
  Trunk port encapsulation: IEEE 802.1q
 It belongs to a link-aggregation
 Master port: GigabitEthernet1/0/11
 Last 300 seconds input:  2 packets/sec 1957 bytes/sec
 Last 300 seconds output:  2 packets/sec 322 bytes/sec
 Input(total):  274612 packets, - bytes
         - broadcasts, - multicasts, - pauses
 Input(normal):  274612 packets, 23868716 bytes
         2463 broadcasts, 3080 multicasts, 0 pauses
 Input:  0 input errors, 0 runts, 0 giants,  - throttles, 0 CRC
         0 frame,  0 overruns, 0 aborts, - ignored, - parity errors
 Output(total): 540857 packets, - bytes
         - broadcasts, - multicasts, - pauses
 Output(normal): 540857 packets, 669171676 bytes
         11800 broadcasts, 99165 multicasts, 0 pauses
 Output: 0 output errors,  - underruns, - buffer failures
         0 aborts, 0 deferred, 0 collisions, 0 late collisions
         - lost carrier, - no carrier

Not quite sure what a "master port" was, I went to ask wise old Google. It came back with links that referenced 3Com link-aggregation ports and the best I could discern was that when a link-aggregation group is created, a master port is chosen by the lowest port number in the group and all traffic is sent through that port and the others are used for "backup".

This bummed me out a little and I spent the next hour playing with LACP and this link-aggregation group setting to see if I could make it work. Nothing... I could not get my ESXi server to send or receive data on both ports at the same time no matter what I did. Last resort I did what any good network engineer would do:

[4200G] ?

I'm not quite sure of what I was looking for but I came across a curious command:

port-group

Huh? 

[4200G]port-group ?
  INTEGER<1-100>  Specify port-group ID

[4200G]port-group 1 

With a ? I can see that I can add ports to this port group. I asked Google again what this "port-group" command was but there was a big fat zero hits on it. Eventually I just figured it couldn't hurt to try it so I blew up the link-aggregation group and threw my two ports into this new "port-group" and logged into each of my server to do a transfer from a NAS to test. 
 
[4200G] undo link-aggregation group 1
[4200G] interface GigabitEthernet 1/0/11
[4200G-GigabitEthernet1/0/11] port-group 1
[4200G-GigabitEthernet1/0/11] interface GigabitEthernet 1/0/12
[4200G-GigabitEthernet1/0/12] port-group 1
[4200G-GigabitEthernet1/0/12]

Here's the results... Check it out:

"Houston, we have lift off". I couldn't believe it. Not only is there NO reference to this command anywhere in the manuals that I've read but it seems to be mimicking Cisco proprietary non-LACP etherchannel. I'm still pretty stunned and happy to say the least.

Lab time! Part 2

So here I am with three sweet servers for $600. I now had another issue to solve...

How do I use them?

Funny as it may sound, something I do everyday at work slipped my mind, before my wife asked me how much was my lab going to cost us overall when I mentioned buying the servers. I then sat down to think about what my design goal was and here's what I came up with:

• With three servers I can dedicate one to running my home server infrastructure (ie. AD, DNS, etc.)
• I can run my VMware lab on the other two by virtualizing three ESXi servers on each of those hosts
• I wanted to use shared storage from the get-go just to get more practice in and to test some theories out (like running VDI completely off of NFS since vSphere 5.0 and above can reference the storage device by DNS name now... not exactly sure how View composer is going to react to that, but we'll see). Just needed to pick out a NAS that supports VAAI
• On the physical side, I need to completely enclose these servers in some kind of sound proof rack so I wouldn't drive myself and my wife crazy when we're up here in my office trying to relax.
• I would need a gigabit switch with jumbo frames and port channeling (LACP) for decent iSCSI performance
• I would need a good UPS with at least voltage regulation and hopefully of the pure sine wave type

I settled on using a Synology DS1512+ for the shared storage because of the VAAI support, dual nics, five bays and nice reviews (http://www.smallnetbuilder.com/nas/nas-reviews/31774-synology-ds1512-disk-station-reviewed). I'm also going to use a TP-Link TLSG2216 to run the network because of the low cost (roughly $150), LACP support and jumbo frames. From a high-level standpoint, this is what my lab will look like:

(on a side note... yes it does say 2.5GHz for the processor speeds on that picture. That's because I'm still on the fence about picking up some cheap low power Xeon L5420's to swap in there)

Next up... how I will keep my sanity while using the lab.

Lab time! Part 1

I've been wanting to start writing about my adventures in home lab building but it's been difficult to come up with a way to talk about it that hasn't been hashed out one way or another before somewhere else. I've lost count of the number of web pages and forum threads that are devoted to building a VMware home lab so instead of rehashing all their posts here I think I'm just going to write about how I'm building mine, what I'm looking for design wise, and how much I'll be spending on each item. I hope these next series of posts will help someone else out as well.

My first objective of this home lab project was overall conceptual design. How was my lab going to look at from a high level overview and how was it going to function the way I was hoping it would. Since I don't have access to an actual lab at my work, I needed to build something that could not only act as a host for my own servers but that I could use to VPN into and test work stuff out on as well. I  knew from the beginning that one server wasn't going to do it for me... even with the ability to virtualize ESXi on top of itself. Basically it boiled down to I didn't have or want to spend $1500 to $2000 on a few machines that might cost me a bunch just to keep them running. I messed around with the possibility of building a AMD 8-core Bulldozer based "server" for a while because I could find a few deals that had the processor and board together for about $250 (namely at MicroCenter) but every time I came up with a price, I was still looking at around $500 - $700 for just 1 host with 32GB of RAM and some add-ons that I was specifically looking for. Since I'm a scrooge with my own cash I kept looking for a better deal. After months of searching for the right gear, I hopped onto ebay one day and saw this puppy:

I'll have to admit that after sitting there thinking about it I got a little excited. Dual quad core low- power 64 bit Xeons (with VT-d support) and 24GB of DDR2 FB-DIMMs with a 160GB hard drive, dual on-board gigabit NICs and a tiny 280W power supply... and it was half size. Every other time I was searching for cheap servers the usual batch of them came up with 4 to 8GB of DDR2, full size, 120W TDP processors that would cost me an arm and a leg to run and when I went to price it out, the extra RAM totaled four times the cost of the server itself. Not only that, but I know how servers were built during the days of socket 771... Loud. Very loud. These servers that I wanted needed to be housed in my office which also served as a night time get-away for my wife after we put the toddler to bed since we have a couch and TV in here. 

On a side note: Let's be honest here for a second... I'm not really sure why everyone wants hardware pass-through on their home server. Granted, you can do some really tricky storage related things with it but for all intents and purposes, if you're building a lab to mimic a real world scenario and to study for VMware certs, is it that important? Back to the fun stuff...

Now, I was hoping to go max RAM on VMware Hypervisor but for the price I was getting these servers for, I wasn't going to bitch about being only 8GB short per server. To be honest as well, picking up three of these for $600 and some change was a steal. I had an entire three host cluster for the price of one newer computer. I also told myself I would find a way to make these guys run quiet and I would have to do it without breaking the bank.

Stay tuned to see how I accomplished this feat!

View 5.1 certificate for multiple connection servers

I apologize for the delay with posts. I've been pretty busy at work keeping everything ship-shape. I finally got to a point where I could start thinking about some redundancy in our View environment and after going through some documents and other blogs, I decided to go with two load balancers on the outside connected to two security servers which in turn have two dedicated connection servers on the inside. Then on the inside I have two dedicated connection servers and two load balancers (btw, I've been testing out Zen Loadbalancers internally and so far, for open source, this program is pretty impressive - and it's still being developed!)

The one thing that had me a little stumped was the SSL certificate. I was going from one connection server and one security server to four connection servers and two security server. My original design was for view.<organization>.com to be reachable no matter where you were - inside or out and I wanted to keep it that way. Well, the problem was that the certificate I purchased was of the single domain variety for around $40 from GoDaddy. I loaded that cert on both the original security server and connection server and had my internal DNS doing eveything for both the servers. That worked out just fine... until now. I didn't want to ask to purchase a multi-domain SSL cert and I didn't want to risk re-keying the original with a CSR that had multiple subject alternative names (SANs) due to the fact that GoDaddy's support site said you had to purchase a multi-domain SSL to use SANs in the first place.

Well the answer it turns out, was our internal CA server. I kept the GoDaddy cert for the outside servers and installed a new internal cert on our internal connection servers. As it's being tested out now, everything is working just fine (just need to convert and upload the CA's root certificate to the zero clients and I should be good to go). Since it's not clearly documented anyways on how to actually create a multi-domain CSR, I thought I'd document the process here. This process is for a Windows Server 2008R2 machine.

Step 1) Enter MMC and open the certificates snap in

Step 2) Right-click, go to "All Tasks" --> "Advanced Options" --> "Create Custom Request"

Step 3) Start the enrollment process

Step 4) Select "Proceed without enrollment policy"

Step 5) Under "Template" select "Legacy Key" and keep "PKCS #10" selected under "Request Format"

Step 6) Under "Custom Certificate" hit "Properties"

Step 7) Under the "General Tab" type in "vdm" as the "Friendly Name"

Step 8) On the "Subject" tab, select the drop down for "Subject Name" and select "Organization". Type in the FQDN of the DNS entry point for your internal View clients (ie. view.yourdomain.com) and hit "Add". Then for the drop down menu under "Alternative Name", select "DNS" and being by entering the FQDN of the DNS entry point

Step 9) Then add a "DNS" entry under "Alternative Name" for each or your connection servers (this is vitally important if you're going to load balance connection servers).

Step 10) On the "Extensions" tab, under "Key Usage", add "Digital signature" and "Key encipherment"

Step 11) Then under "Extended Key Usage" select and add "Server Authentication" and "Client Authentication"

Step 12) On the "Private Key" tab and under "Cryptographic Service Provider", have only "Microsoft Strong Cryptographic Provider (Signature)" and "Microsoft RSA SChannel Cryptographic Provider (Encryption)" selected

Step 13) Under the "Key Options" section, choose your key strength. I chose 2048 and marked the key as exportable in the example.

Step 14) Click "OK" and it should bring you back to this screen

Step 15) Hit "Next" and select where you want to save the CSR. Make sure "Base 64" is marked as the format.

Congrats! You now have a CSR for your load balanced View environment created with Windows Server 2008R2. The next step is up to you, but I chose to have the cert signed with our internal CA to save some money. Getting a multi-domain certificate can be a little expensive. One of the drawbacks to having an internal CA signed certificate is you have to load the root CA on mobile devices but depending on your MDM platform, that could be an automatic thing for you.

I'm still here

Two months.

That's almost an eternity in the online world (and quite frankly, the real world as well if you've been as busy as I have). To tell you the truth, I've been a little nervous to write this post as sometimes I feel like there's just not enough time to actually write down everything that's happened. Just in the small amount of time that I've tried this blogging thing, I've grown an immense gratitude toward everyone that has the propensity to sit down and actually crank out post after post every two to three days... AND have something original to write about (or at least something nice to say). I really do want to have this blog work and maybe when my real work slows down a bit I can get a little bit deeper into writing a bit more. I just really wanted to stop by and say that I haven't abandoned this blog (not that anyone actually reads it right now anyways) and hopefully in the near future I'll have a more meaningful post for you... Like the process of building up my home lab - which I'm really psyched about!

Talk to you soon