Turning an MSNTV 2 Into Something Useful

So I picked up this little guy from a CompUSA clearance rack for something like $20. I had read about some guy who hacked his MSNTV 2 to run Linux and it seemed like something fun to do if I were ever bored. Well I downloaded all the necessary files except for the most significant one, the hacked BIOS image. He claimed he couldn't post it for fear of getting sued (I don't blame him, this is Microsoft) so I searched high and low for it. After a few weeks I gave up and shelved it.

Not too long ago a friend of mine showed me the wonders of coreboot (formerly LinuxBIOS). These crazy guys from the Cluster Research Lab needed a way to quickly boot up a cluster node and got fed of with all the nonsense that the traditional PC BIOS tended to have and decided that they'd write their own BIOS using..what else? Linux. Since they opened the source up it relatively quickly became a viable BIOS for a number of boards. The board in the MSNTV 2 (RM4100) happens to be one of those boards. Whoo hoo!

So if you don't know me personally you won't know that I tend to start on projects and then get bored before they're finished, which kinda sucks, but that's just who I am. The purpose of writing about this therefore is to 1) Document the process (although really I'm just following the awesome documentation that I found at settoplinux.org) and 2) Force myself to finish this project in the event someone is actually reading this. ;-)

How to Force Collection of Interface Stats in NMIS

So today I figured out how to explicitly set an interface alias in NMIS. We have a Force10 C300 switch and NMIS was able to discover all the interfaces via SNMP just fine. The problem was that NMIS is configured to only collect information from interfaces that have the description set. Even though we set the description on our interfaces, for some reason it wasn't treating it the same way as, for instance, a Cisco switch. In any case, after a few minutes of reading the documentation I figured out that the NMIS author already thought of this potential problem and supplied a work-around. It's called sysInterface. Basically, you populate a tab-delimited CSV file with entries like this:

# THE FIRST LINE NON COMMENT IS THE HEADER LINE AND REQUIRED

Description Node Overwrite_Description ifIndex ifSpeed
192.168.0.1 ^f10-c300$ 1 100975616 null
192.168.0.2 ^f10-c300$ 1 107529216 null

So the node is a regular expression of the switch (or router or whatever) you want it to act on, as is the ifIndex. I just snmpwalked the switch to figure out all the indexes and came out with a list of switch port numbers (like GigabitEthernet 0/0) and OID indexes. Then I cross-referenced that with another list I had with the switch port numbers and the host IP address to come up with the final CSV. The Overwrite_Description field is a boolean, set it to 1 (or maybe even true, I haven't tried that) and it will activate the interface description overrride. Leave ifSpeed blank (or null rather) as it will just get the correct value from SNMP. Also, if you don't have the list of host names or IP addresses you want to use for the Description field, just use the keyword ifDescr and it will just copy whatever the "hardware label" is (that's like GigabitEthernet 0/0 or whatever).

So once you do that, you just run nmis.pl type=update debug=true and you should see the overwritten ifAliases and collect will be forced to true. Yay.

Updating MySQL Passwords When Using old_passwords

So I learned this awhile back but forgot tonight while making a change on a production system. No real harm, just broke the site for a couple minutes while I quickly scanned through the MySQL documentation. Here's the scenario:

MySQL version 4.1
PHP version 5.2.5

We had some legacy PHP code that required us to use the old_passwords option in my.cnf. Instead of upgrading all our code to use 32-bit password hashes we just enabled old_passwords instead. Yes, this is more insecure, but I have to pick my battles.

Anyway, what happened tonight was that I needed to update the mysql password for one of our applications that was configured to use 16-bit hashes (the old password format). I issued this:

UPDATE mysql.user SET Password = PASSWORD('SomeUnguessablePassword') where User = 'galileo';

This updated the password for that user, but the application broke. I tried connecting to it from the MySQL client and it failed. I'm not sure why I thought of the old password format at that point...call it a stroke of good luck...but I quickly searched for old_passwords in the MySQL documentation and was reminded of the OLD_PASSWORD() function. So I then issued the following:

UPDATE mysql.user SET Password = OLD_PASSWORD('SomeUnguessablePassword') where User = 'galileo';

And voila! We were back in business.

One tip for those that are changing passwords and want an easy way to backout (this is mostly for the noobs). SELECT the current password from the user table and have this query ready just in case something breaks.

UPDATE mysql.user SET Password = '77da416361d244c3' where User = 'galileo';

Using the hashed password you got previously of course. That way, even if you didn't know what the plain text password was you could still revert back.

Rewriting URLs on the Citrix Netscaler

So today I had the task of setting up a failover configuration for our email deployment server, affectionately known as "Strongbad".

We created a VM to function as the failover server, but in order to get things to work we needed to modify the URL to capture the entire path and pass it as a value to a PHP script. For instance, the URL the user would click on would look like:

http://strongbad.company.com/track?type=click&mailingid=42&messageid=42&databaseid=test&serial=20090203&emailid=sb@strongbad.com&userid=1&extra=&&&2004&&&http://tracking.company.com/account/listings/?emailUserId=42

If the Strongbad box were to become unavailable, we want the VM to step in. There is a script called strongbad_redirect.php and to simplify things we're just passing the original path directly to this script. The script takes care of parsing out the data and redirecting the user to the appropriate page.

The pertinent lines in the Netscaler are:

add rewrite action act_sbad_req_rewrite insert_before HTTP.REQ.URL.PATH.GET(1) "\"strongbad_redirect.php?\""



add rewrite policy pol_sbad_rewrite "HTTP.REQ.HOSTNAME.SERVER.EQ(\"strongbad.company.com\")" act_sbad_req_rewrite



bind lb vserver mail1 -policyName pol_sbad_rewrite -priority 100 -gotoPriorityExpression END -type REQUEST

Slice and Dice with awk

Just learned a new way to slice and dice a text file. So say you have a file that's 1 million lines long, but you only want to see lines 234567 to 300000. You can use awk to only print out those lines.

awk '{ if (NR >234567 && NR<300000) { print $0; }}' bigfile.txt > bigfile-sliced.txt

Why you shouldn't overclock your GPU


This is an actual picture of a Dell PowerEdge (1850 I think) from my last job. I received a call early in the morning from one of the data center engineers letting me know that they had to pull the power plug because smoke was coming out of the case. Oops.

Poor Pug

I'm not sure why I find this so hilarious but...I do.



Check out more of these at Fail Dogs.

Google Maps API and Trulia



My company's 3 minutes of fame (and I'm in it for a whole 2 seconds - w00t!)

Mystery Tire vs Civic




This happened this past Saturday while we were on our way to get some lunch. I was accelerating on the on-ramp of the freeway near our house when I saw this huge tire come rolling downhill. It was heading towards the shoulder so I kept moving straight, thinking to myself, "Damn, I'm glad I'm going to make it past that thing without hitting it." Alas, it hit something on the side that caused it to ricochet towards us. Tire versus Civic...tire wins.

So I think we've come up with our shopping list after the 2 month evaluation period. I'll try to post some shots of us building the whole data center so y'all can pretend you're also working for one of the funnest (yeah that's a word) companies around. So here's what we're going to get...

Citrix NetScaler 9010

I know it seemed like we were leaning towards the Big-IP, but in truth our evaluation showed that both devices are equally good. The Big-IP has the advantage of being able to use iRules to do some pretty creative things at the site edge before it passes anything back to the servers. The NetScaler had the advantage of very trivial remote web logging setup, software that is very IOS-like and therefore would be easy to learn for any NetAdmin who knows Cisco, and very good word-of-mouth recommendations. This isn't taking anything away from the Big-IPs. I still like them and would recommend them to anyone.

Force10 C300

So I don't know that much about Force10 but I have had some experience administering Cisco 6500 series switches, which this thing presumably is competing against. They look the same, I'm sure the software is pretty similar too. I've heard anecdotal evidence that these outperform the Cisco's, but in reality, anything is going to be better than the POS Dell PowerConnect switches we are currently forced to use now. This C300 has 336 10/100/1000 Mbps Ethernet ports which is perfect for our phase 1 plans (about 60 hosts with 3 network interfaces in use). I'm a NetAdmin-in-training and I actually find it more enjoyable to work with network devices than systems so I really can't wait to get my grubby little paws on this bad boy.




3Par Inserv S400



So I've never heard of 3Par until our Ops lead joined the team. He used to work for 3Par a few years ago and it might seem like we're picking this product over the likes of NetApp, HDS, or EMC because of that reason...but after visiting 3Par's lab and having a few meetings with the other vendors it really became a no brainer decision. 3Par wasn't the cheapest solution, but in terms of performance, ease of use, and customer support they beat everyone else we checked out. This was actually the most controversial (although I use that term very lightly) item when we were discussing the data center project with the Engineering team. We currently have a very distributed, DAS (Direct Attached Storage) environment and it's understandable that they were a bit hesistant when we said that we could move away from all the local hard drives and just use centralize storage. From an administrative standpoint it's a no-brainer. From a performance standpoint, it took some convincing but in the end we all agreed that our systems would perform no lesser (and possibly better) with centralized storage. We'll use DAS for certain situations of course.




Sun Netra X4200


So we decided on Sun to host our site. It was between Sun and HP. Both are great systems (especially compared to Dell) but the clincher for us was remote management capabilities. HP's ILO is great, but would cost us an additional $300 per server to give us what Sun was including in the base price. The sun boxes are very well manufactured. At my last job I administered Linux on Dell PowerEdges for 5 years and the Sun just blew me away when it came to how it was engineered. The Netras will power all our front-end servers.

Sun Fire X4600


And the Fire will power our back-end servers and also host some VMs. This bad boy is the toy that still makes us all giddy like a school girl. Besides the fact that it has 8 dual-core Opterons (and is quad-core ready) and 32 GB of memory (up to 256 GB), it's the most well-engineered machine I've ever laid my hands on. It looks like a solid piece of metal and it feels like it too. We' re going to try running 16 instances of our front-end code (Apache / PHP) and benchmark it. I'll post more info on our results later.

Damn, this dude can sing...

A co-worker / friend of mine, Invisible Programmer, enlightened me with the magic of Guthrie Govan. This kind of reminds me of Steve Vai when I listen to it (but not when I look at him, hehe).

Can anyone think of a better caption?

NetScaler versus Big-IP Bake-off

Introduction

I spent several days with the Citrix NetScaler and F5 Big-IP. I have used Big-IPs in the past, although it was the 4.x software version which is much different than the current 9.x version. This gave me a small advantage with the Big-IP initially. I've administered Cisco devices also, and the NetScaler CLI felt a lot like Cisco IOS, so initially I think both were equally comfortable to administer. When all you need is to be able to create a virtual IP and balance traffic to your back-end servers, both products can do the job easily and for the most part have equal features in that category. I think the Big-IP has the advantage of being able to use iRules to do things at the edge of the site that we may or may not need now, but would be a nice-to-have for future configurations. F5 also has a useful user community site (DevCentral) and Knowledgebase (Ask F5). Citrix has user forums too, but it didn't seem nearly as informative. Most of their site seems to be focused on Xen.

Load Balancer Observations

The NetScaler uses a Java applet driven GUI which is mostly nice, but using Firefox and Apple's JRE introduces some quirky display issues. Sometimes the page is totally blank and you need to refresh which usually kicks you out and makes you log back in.

The Big-IP GUI is driven by JSP pages and felt "quicker" when navigating through different sections.

The NetScaler interface is very easy to navigate and finding things is simple.

The NetScaler Dashboard looks nicer and some useful info. I wish there was a way to drop more graphs on the page.

The embedded Help sections on the F5 are very convenient.

The PDF documentation for the NetScaler is well written and easy to follow. It would have been nice to see some of the info embedded in the interface like the F5 has.

The Big-IP graphs are actually more informative in my opinion. Being able to see connections in real-time is a real plus.

I like the fact that the Big-IP is running a Linux kernel (albeit an old one) and has GNU tools. Makes it easier to administer for a Linux SysAdmin. Also, the documentation is in man pages, so if you want to know everything you need to know about creating a new virtual server, it's as simple as typing 'man virtual'.


The other advantage of a GNU/Linux environment is that you can actually use sed to do things like globally replace a string in your config file to quickly and easily change something site-wide.

There is a good example of how this is useful. I had to change the self IP address of the external interface and make it internal, but it was not obvious through the GUI or even through bigpipe. Editing the bigip_base.conf file was super easy though.

The CLI on the NetScaler (as mentioned above) feels very IOS-like and was pretty easy to go from a Cisco device, to a Foundry device, to the NetScaler without having to think about what you're doing. The Big-IP uses "bp" commands that take a little getting used to, but after an hour of repetition it's equally easy to administer on the command-line. The built-in documentation (man pages, natch) on the Big-IP is very useful. The NetScaler does the IOS "tab-tab" thing to assist when you've lost your way.

The multiple Gigabit interfaces on the Big-IP would be useful for internally used clustered Web apps that need to be on separate subnets. I don't think we're there yet, but it would be nice to have that option.

The NetScaler has a remote web logging feature that is extremely easy to setup. You install an agent on your logging host via RPM and then enable remote logging on the NetScaler and it's done. The Big-IP can probably handle logging via an iRule, however we didn't get to that question when their system engineer was out last time and searching on DevCentral didn't pull up anything readily.

Simple to configure Cacti graphs on Big-IP, there was even a how-to on F5's site. The graphs allow you to monitor connections per second per virtual server (try doing that on the Foundry). The SNMP MIBS are readily available within both the Big-IP and NetScaler GUI.

Both products offer an API solution so we can write custom applications that can interface with the load balancers. The NetScaler offers this within their GUI, along with the documentation. F5 offers it's API called iControl, along with an SDK, both freely available on their site. There's also an iRules editor (Windows only) that is like an IDE for iRules. It's very useful for quickly writing complex iRules from scratch and has a built-in syntax checker. Both APIs use standard XML / SOAP. The NetScaler provides a few examples in different languages to get you started, but F5's community is again useful in providing a dynamic forum where more than just examples are posted.

So we're evaluating the Sun Fire X4600 and Netra X4200 this month, along with the HP ProLiant DL365. The ProLiant and the Netra are pretty decent boxes, and would be excellent workhorses for your front-end app. The X4600 is what we're drooling over however. With the ability to power 8 dual-core Opterons (and quad-core when they're available), and 256 GB of memory, this baby is begging for virtualization. It's nice that we're also evaluating both Citrix XenServer and VMware ESX 3.5 this month too. Whoo hoo!

The Fire X4600 has a very nice modular design. You can buy it with 2 to 8 modules and each module holds a dual-core processor and up to 32 GB of memory. Ours only has 8 dual-core procs and 32 GB's of memory total, but that is more than enough for what we're planning on testing it with. Here's a closer look at the module.



That big copper heatsink sits on top of the CPU.

I'm not quite sure what this guy was thinking, but clearly Canadians are crazy.

So a friend introduced me to the magical genius that is Mark "Bill" Bailey. If you haven't seen any of his stand-up comedy, here's a taste of what he does...brilliant!



You can buy his CD's and DVD's at his official online store.

My Dad just sent me a link to a YouTube video of this amazing R/C helicopter pilot doing a run. The video speaks for itself.




I remember my dad buying me an R/C helicopter (two in fact, as one had an unfortunate accident) when I was a kid. It was one of the most exhilarating times I've ever had growing up. I was a horrible pilot because I knew how much the thing cost and was petrified with fear that I'd crash it. The fear mixed with excitement like nitro and methane every time I'd hear that little motor start up. I've been to a couple of R/C helicopter events since then (although I've never seen anyone as crazy as this Alan Szabo Jr guy). If you haven't seen one in person, I highly recommend checking one out.

XenSource v4 XenServer

So we got our 1 month eval license for XenServer and I just installed it on this Dell PowerEdge 2950 with dual quad-cores, 8 GB of RAM, and about 1.5 TB's of storage we had lying around. The install of XenServer was pretty boring: click, click, click, reboot, done (which, of course, is a good thing). The thing I really wanted to check out was their XenCenter Management client. The one bad thing is that it needs to run on Windows. I have a virtualized XP VM running via Parallels on my MacBook so it wasn't too big of a deal I guess.

The XenCenter interface looks like a typical Microsoft MMC plug-in. It was pretty simple to figure out how to do basic things like create new VMs, create Storage Repositories (this is an NFS share on the XenServer and I'm planning on storing all the VMs here to play with XenMotion), check the performance stats for the XenServer and its guests, configure network interfaces, etc. I have to say, so far this is looking pretty good for Xen. That was my one big beef with Xen versus VMware...their GUI was practically non-existent. I know some of you hardcore UNIX guys are scoffing right now. I don't mind the command-line...for most purposes I'll probably end up using it exclusively...but GUIs have their place like when you have multiple XenServers running multiple XenGuests and you want to do things to a subset of hosts.

I'm going to create several VMs (like 6 web servers and 2 database servers) and then load our site on them. Then we'll put them behind the load balancers and maybe run some load tests. It'd be nice to see how easily we can XenMotion VMs off of one physical server to simulate some kind of scheduled maintenance, and what impact a user would experience during the process.

Citrix NetScaler

So today the eval NetScaler arrived and the Citrix engineer went over some basic setup info with us, then turned it over into our eager hands. It's going to be really fun toying around with both the Big-IP and the NetScaler to see which one we like.

Performance isn't really that big of an issue for us (yet). We Akamaize a lot of our static content (to the tune of offloading about 40 Mbps of traffic on average). The main question is, can we SysAdmins figure out how to do what we need to do easily, and as quickly as possible? We want to see an easy to use CLI *and* a functional GUI. We also want to see statistics and detailed reporting, as well as the ability to create graphs in Cacti for as much useful data as possible.

The NetScaler quote came in at around $50,000 for a pair (we want redundancy of course). The Big-IP came in at about $35,000 a pair. We're not quite sure why the difference is so much, but hopefully after several days of tinkering and some help from our VAR, we'll get to the bottom of that.