For my clustering solutions, I’ve chosen live cd’s (clusterknoppix to be specific). I purchased about $2500 in old hardware (Pentium 850 Dual Core’s with 1GB of RAM running at 2.66ghz per core), and combining that with my existing hardware and I end up with a cluster with 15 cores running combined at a little over 40ghz and 6GB of ram. It’s not too shabby, if I do say so myself. It is an openMosix based cluster, so the load balancing is awesome. I’ve been using it thus far to do some prime number factoring.

One of the nice things about mosix clustering is its ease of use, while one of the nice things about beowulf clustering is the ease of data sharing and segregation. I’ve discussed previously in GA In Parallel, some more interesting thoughts… that load balancing is a problem with Beowulf clusters, and even with my crazy scheme of Multi-Level Hybrid Clusters (MLHC). The basic idea behind MLHC is to create a beowulf cluster whose slave nodes are actually mosix clusters. It was an invention more of necessity than anything else, but it solved my problem nonetheless.

My plan now is to install LAM/MPI (beowulf software, essentially) onto one of my mosix nodes and then create all of the MPI processes on a single node and allow the mosix kernel to load-balance the cluster. At that point, I’ll have a load-balanced beowulf cluster. My main issue until now has been creating multiple instances of complete populations of my GANet software, because a single member of a population can be up to 100mb. Since some of my nodes have only 256mb of RAM, they can’t even run one instance of GANet with a decent sized population. The solution then is to create multiple small populations that share information, with each population being small enough to run on a single node. Also, I’ve learned that my population members don’t need to be quite as large as I’m making to start, especially considering that I allow them to grow dynamically.

Today’s lesson, beowulf is nice, but mosix load-balances. So take both.