Why you shouldn't overclock your GPU


This is an actual picture of a Dell PowerEdge (1850 I think) from my last job. I received a call early in the morning from one of the data center engineers letting me know that they had to pull the power plug because smoke was coming out of the case. Oops.

Poor Pug

I'm not sure why I find this so hilarious but...I do.



Check out more of these at Fail Dogs.

Google Maps API and Trulia



My company's 3 minutes of fame (and I'm in it for a whole 2 seconds - w00t!)

Mystery Tire vs Civic




This happened this past Saturday while we were on our way to get some lunch. I was accelerating on the on-ramp of the freeway near our house when I saw this huge tire come rolling downhill. It was heading towards the shoulder so I kept moving straight, thinking to myself, "Damn, I'm glad I'm going to make it past that thing without hitting it." Alas, it hit something on the side that caused it to ricochet towards us. Tire versus Civic...tire wins.

So I think we've come up with our shopping list after the 2 month evaluation period. I'll try to post some shots of us building the whole data center so y'all can pretend you're also working for one of the funnest (yeah that's a word) companies around. So here's what we're going to get...

Citrix NetScaler 9010

I know it seemed like we were leaning towards the Big-IP, but in truth our evaluation showed that both devices are equally good. The Big-IP has the advantage of being able to use iRules to do some pretty creative things at the site edge before it passes anything back to the servers. The NetScaler had the advantage of very trivial remote web logging setup, software that is very IOS-like and therefore would be easy to learn for any NetAdmin who knows Cisco, and very good word-of-mouth recommendations. This isn't taking anything away from the Big-IPs. I still like them and would recommend them to anyone.

Force10 C300

So I don't know that much about Force10 but I have had some experience administering Cisco 6500 series switches, which this thing presumably is competing against. They look the same, I'm sure the software is pretty similar too. I've heard anecdotal evidence that these outperform the Cisco's, but in reality, anything is going to be better than the POS Dell PowerConnect switches we are currently forced to use now. This C300 has 336 10/100/1000 Mbps Ethernet ports which is perfect for our phase 1 plans (about 60 hosts with 3 network interfaces in use). I'm a NetAdmin-in-training and I actually find it more enjoyable to work with network devices than systems so I really can't wait to get my grubby little paws on this bad boy.




3Par Inserv S400



So I've never heard of 3Par until our Ops lead joined the team. He used to work for 3Par a few years ago and it might seem like we're picking this product over the likes of NetApp, HDS, or EMC because of that reason...but after visiting 3Par's lab and having a few meetings with the other vendors it really became a no brainer decision. 3Par wasn't the cheapest solution, but in terms of performance, ease of use, and customer support they beat everyone else we checked out. This was actually the most controversial (although I use that term very lightly) item when we were discussing the data center project with the Engineering team. We currently have a very distributed, DAS (Direct Attached Storage) environment and it's understandable that they were a bit hesistant when we said that we could move away from all the local hard drives and just use centralize storage. From an administrative standpoint it's a no-brainer. From a performance standpoint, it took some convincing but in the end we all agreed that our systems would perform no lesser (and possibly better) with centralized storage. We'll use DAS for certain situations of course.




Sun Netra X4200


So we decided on Sun to host our site. It was between Sun and HP. Both are great systems (especially compared to Dell) but the clincher for us was remote management capabilities. HP's ILO is great, but would cost us an additional $300 per server to give us what Sun was including in the base price. The sun boxes are very well manufactured. At my last job I administered Linux on Dell PowerEdges for 5 years and the Sun just blew me away when it came to how it was engineered. The Netras will power all our front-end servers.

Sun Fire X4600


And the Fire will power our back-end servers and also host some VMs. This bad boy is the toy that still makes us all giddy like a school girl. Besides the fact that it has 8 dual-core Opterons (and is quad-core ready) and 32 GB of memory (up to 256 GB), it's the most well-engineered machine I've ever laid my hands on. It looks like a solid piece of metal and it feels like it too. We' re going to try running 16 instances of our front-end code (Apache / PHP) and benchmark it. I'll post more info on our results later.

Damn, this dude can sing...

A co-worker / friend of mine, Invisible Programmer, enlightened me with the magic of Guthrie Govan. This kind of reminds me of Steve Vai when I listen to it (but not when I look at him, hehe).

Can anyone think of a better caption?

NetScaler versus Big-IP Bake-off

Introduction

I spent several days with the Citrix NetScaler and F5 Big-IP. I have used Big-IPs in the past, although it was the 4.x software version which is much different than the current 9.x version. This gave me a small advantage with the Big-IP initially. I've administered Cisco devices also, and the NetScaler CLI felt a lot like Cisco IOS, so initially I think both were equally comfortable to administer. When all you need is to be able to create a virtual IP and balance traffic to your back-end servers, both products can do the job easily and for the most part have equal features in that category. I think the Big-IP has the advantage of being able to use iRules to do things at the edge of the site that we may or may not need now, but would be a nice-to-have for future configurations. F5 also has a useful user community site (DevCentral) and Knowledgebase (Ask F5). Citrix has user forums too, but it didn't seem nearly as informative. Most of their site seems to be focused on Xen.

Load Balancer Observations

The NetScaler uses a Java applet driven GUI which is mostly nice, but using Firefox and Apple's JRE introduces some quirky display issues. Sometimes the page is totally blank and you need to refresh which usually kicks you out and makes you log back in.

The Big-IP GUI is driven by JSP pages and felt "quicker" when navigating through different sections.

The NetScaler interface is very easy to navigate and finding things is simple.

The NetScaler Dashboard looks nicer and some useful info. I wish there was a way to drop more graphs on the page.

The embedded Help sections on the F5 are very convenient.

The PDF documentation for the NetScaler is well written and easy to follow. It would have been nice to see some of the info embedded in the interface like the F5 has.

The Big-IP graphs are actually more informative in my opinion. Being able to see connections in real-time is a real plus.

I like the fact that the Big-IP is running a Linux kernel (albeit an old one) and has GNU tools. Makes it easier to administer for a Linux SysAdmin. Also, the documentation is in man pages, so if you want to know everything you need to know about creating a new virtual server, it's as simple as typing 'man virtual'.


The other advantage of a GNU/Linux environment is that you can actually use sed to do things like globally replace a string in your config file to quickly and easily change something site-wide.

There is a good example of how this is useful. I had to change the self IP address of the external interface and make it internal, but it was not obvious through the GUI or even through bigpipe. Editing the bigip_base.conf file was super easy though.

The CLI on the NetScaler (as mentioned above) feels very IOS-like and was pretty easy to go from a Cisco device, to a Foundry device, to the NetScaler without having to think about what you're doing. The Big-IP uses "bp" commands that take a little getting used to, but after an hour of repetition it's equally easy to administer on the command-line. The built-in documentation (man pages, natch) on the Big-IP is very useful. The NetScaler does the IOS "tab-tab" thing to assist when you've lost your way.

The multiple Gigabit interfaces on the Big-IP would be useful for internally used clustered Web apps that need to be on separate subnets. I don't think we're there yet, but it would be nice to have that option.

The NetScaler has a remote web logging feature that is extremely easy to setup. You install an agent on your logging host via RPM and then enable remote logging on the NetScaler and it's done. The Big-IP can probably handle logging via an iRule, however we didn't get to that question when their system engineer was out last time and searching on DevCentral didn't pull up anything readily.

Simple to configure Cacti graphs on Big-IP, there was even a how-to on F5's site. The graphs allow you to monitor connections per second per virtual server (try doing that on the Foundry). The SNMP MIBS are readily available within both the Big-IP and NetScaler GUI.

Both products offer an API solution so we can write custom applications that can interface with the load balancers. The NetScaler offers this within their GUI, along with the documentation. F5 offers it's API called iControl, along with an SDK, both freely available on their site. There's also an iRules editor (Windows only) that is like an IDE for iRules. It's very useful for quickly writing complex iRules from scratch and has a built-in syntax checker. Both APIs use standard XML / SOAP. The NetScaler provides a few examples in different languages to get you started, but F5's community is again useful in providing a dynamic forum where more than just examples are posted.

So we're evaluating the Sun Fire X4600 and Netra X4200 this month, along with the HP ProLiant DL365. The ProLiant and the Netra are pretty decent boxes, and would be excellent workhorses for your front-end app. The X4600 is what we're drooling over however. With the ability to power 8 dual-core Opterons (and quad-core when they're available), and 256 GB of memory, this baby is begging for virtualization. It's nice that we're also evaluating both Citrix XenServer and VMware ESX 3.5 this month too. Whoo hoo!

The Fire X4600 has a very nice modular design. You can buy it with 2 to 8 modules and each module holds a dual-core processor and up to 32 GB of memory. Ours only has 8 dual-core procs and 32 GB's of memory total, but that is more than enough for what we're planning on testing it with. Here's a closer look at the module.



That big copper heatsink sits on top of the CPU.

I'm not quite sure what this guy was thinking, but clearly Canadians are crazy.

So a friend introduced me to the magical genius that is Mark "Bill" Bailey. If you haven't seen any of his stand-up comedy, here's a taste of what he does...brilliant!



You can buy his CD's and DVD's at his official online store.

My Dad just sent me a link to a YouTube video of this amazing R/C helicopter pilot doing a run. The video speaks for itself.




I remember my dad buying me an R/C helicopter (two in fact, as one had an unfortunate accident) when I was a kid. It was one of the most exhilarating times I've ever had growing up. I was a horrible pilot because I knew how much the thing cost and was petrified with fear that I'd crash it. The fear mixed with excitement like nitro and methane every time I'd hear that little motor start up. I've been to a couple of R/C helicopter events since then (although I've never seen anyone as crazy as this Alan Szabo Jr guy). If you haven't seen one in person, I highly recommend checking one out.

XenSource v4 XenServer

So we got our 1 month eval license for XenServer and I just installed it on this Dell PowerEdge 2950 with dual quad-cores, 8 GB of RAM, and about 1.5 TB's of storage we had lying around. The install of XenServer was pretty boring: click, click, click, reboot, done (which, of course, is a good thing). The thing I really wanted to check out was their XenCenter Management client. The one bad thing is that it needs to run on Windows. I have a virtualized XP VM running via Parallels on my MacBook so it wasn't too big of a deal I guess.

The XenCenter interface looks like a typical Microsoft MMC plug-in. It was pretty simple to figure out how to do basic things like create new VMs, create Storage Repositories (this is an NFS share on the XenServer and I'm planning on storing all the VMs here to play with XenMotion), check the performance stats for the XenServer and its guests, configure network interfaces, etc. I have to say, so far this is looking pretty good for Xen. That was my one big beef with Xen versus VMware...their GUI was practically non-existent. I know some of you hardcore UNIX guys are scoffing right now. I don't mind the command-line...for most purposes I'll probably end up using it exclusively...but GUIs have their place like when you have multiple XenServers running multiple XenGuests and you want to do things to a subset of hosts.

I'm going to create several VMs (like 6 web servers and 2 database servers) and then load our site on them. Then we'll put them behind the load balancers and maybe run some load tests. It'd be nice to see how easily we can XenMotion VMs off of one physical server to simulate some kind of scheduled maintenance, and what impact a user would experience during the process.

Citrix NetScaler

So today the eval NetScaler arrived and the Citrix engineer went over some basic setup info with us, then turned it over into our eager hands. It's going to be really fun toying around with both the Big-IP and the NetScaler to see which one we like.

Performance isn't really that big of an issue for us (yet). We Akamaize a lot of our static content (to the tune of offloading about 40 Mbps of traffic on average). The main question is, can we SysAdmins figure out how to do what we need to do easily, and as quickly as possible? We want to see an easy to use CLI *and* a functional GUI. We also want to see statistics and detailed reporting, as well as the ability to create graphs in Cacti for as much useful data as possible.

The NetScaler quote came in at around $50,000 for a pair (we want redundancy of course). The Big-IP came in at about $35,000 a pair. We're not quite sure why the difference is so much, but hopefully after several days of tinkering and some help from our VAR, we'll get to the bottom of that.

3PAR

So we went to 3Par Headquarters today and got our hands dirty with their storage systems. Overall I was pretty impressed. These guys know what they're talking about. It's refreshing to find a storage company that's focused on...storage. There are other high end storage companies that either get bored, or get greedy, or get stupid, but they start offering software and other products that, although they might be great products, have nothing to do with storing data. 3Par only has 3 models, but really they have 2 products (the E class and the S Class). The E Class is what we're looking at to start. It's no slouch though. It will store up to 96 TB of data and is wicked fast. They make it incredibly easy to create volumes, change RAID type (online), dynamically optimize the entire system, etc. either through their Java GUI (yuck) or their powerful CLI. Their system can use both Fibre Channel and SATA (they call it Nearline) drives, and you can mix and match as you please. They also invented "thin provisioning" of volumes which means you can export a volume to a host and the host will see, say, 100 GB's, but it's not really using all that space in the beginning. It gives out storage to the host little by little (I think the default is 30 GBs or something) but the host never knows. So if your users say they absolutely need 100 GB's of space, but you're pretty certain they don't, no need to argue. TP to the rescue!

Anyway, it was a very informative visit. We have Citrix coming in tomorrow to drop off an eval NetScaler. So next week I'll write about our experiences with using both the Big-IP and the NetScaler. Stay tuned.

Vendor Evaluations

This month (and possibly running into next month) we're going to be evaluating different vendors for servers, load balancers, storage systems, data centers, and other cool stuff. Oh yeah, I work over at Trulia now as a Systems Engineer.

For servers we're checking out what Sun and HP have to offer. Having had my fair share of Dells, I'm happy for the change. The main thing we're looking for is good remote management. Both HP and Sun seem to know what they're doing in this area.

For load balancers we're checking out F5 Big-IPs and Citrix NetScalers. I've used the F5 back at MarkMonitor so I feel comfortable with them. But I always hear great things about the NetScalers so it's going to be fun checking them both out.

For storage systems we're checking out NetApp, Hitachi, and 3Par. I've used NetApps before and so the familiarity factor is there. We're going to the 3Par facility in Fremont tomorrow so I'm excited about getting my hands on some hundred thousand dollar hardware.

For data centers we're now down to either Equinix or NaviSite. NaviSite is within walking distance from our office, but Equinix is looking like the price winner (and Equinix is a pretty trusted name in the data center world so I feel pretty comfortable about that).

So stay tuned for some unbiased opinions on the above vendors.