From: James Rogers (jamesr@best.com)
Date: Wed Apr 25 2001 - 12:39:48 MDT
At 09:53 AM 4/25/2001 -0700, Dani Eder wrote:
>[...snip...]
>I'm curious how this type of Linux cluster compares
>to the processor requirements for a Web-Brain
>implementation or an estimated size for a seed AI
>(if anyone has taken a stab at that yet).
>[...snip...]
>The cost breakdown is as follows. Note the cluster
>has 64 compute nodes + 2 supervisory nodes. Costs
>are current ones from pricewatch.com:
As a general comment, you don't seem to be considering that these types of
clusters are tuned to solve very specific problems. Their solution isn't
necessarily your solution. The choice of hardware is not arbitrary;
interconnects, RAM, switching, and proc/mobo hardware is all carefully
chosen to maximize throughput for the problem in question. A seemingly
trivial mis-design in your cluster can cost you an order of magnitude in
performance.
Some problems can be solved very cheaply in clusters, with almost perfect
linear scaling (and occasionally, the rare super-linear scaling). Other
types of problems demonstrate logarithmic scaling, even with very expensive
interconnects. I expect the AI problem falls somewhere in the middle,
being neither embarrassingly parallel nor excruciatingly serial.
>1.33 GHz Athlon CPU: $227 x 66 units
Excellent choice for clusters in general -- the bang/buck king.
>256 MB PC133 SDRAM: $51 x 66 units
Just a practical word of warning about RAM: reliable RAM can make or break
your cluster, and it is usually considered to be a good idea to pay a
little more to get better RAM. Why? Because when you have several Gb of
RAM under heavy use for days on end, you *will* get the occasional bit-flip
error or RAM behavior anomaly. In computational clusters, this frequently
manifests itself as anomalous computational results, not just system
failures, and can cause very subtle problems that can take a long time to
fully manifest. Very bad ju ju, and generally more expensive than paying
for better RAM in the long run.
>Athlon Motherboards: $80 x 66 units
The chipsets on the board can make a substantial difference as far as I/O
and memory performance go. Know what you are buying.
>24 port x 100 MB/s ethernet hubs: $210 x 15 units
>LAN cards : $5 x 66 units
This is inadequate as an interconnect fabric unless your problem is
embarrassingly parallel.
1.) Buy a *switch*, not a hub, and make sure that it has a high-bandwidth
low-latency bus.
2.) Which LAN cards you use can be very important, especially under
load. The difference between very good (e.g. 3C9xx and Tulip) and bad
(e.g. RealTek and Intel) chipsets is substantial.
3.) If you are doing any type of problem that is not embarrassingly
parallel (specifically, anything that has dependencies scattered across the
cluster), you'll very likely want faster node interconnects i.e.
channel-bonded 100TX, Gigabit Ethernet, or Myrinet.
4.) For applications that are network intensive, it is highly advisable to
use dual processor motherboards. Why? Because high-throughput network
traffic will generate so many interrupts that your CPU won't get any work
done. On a dual processor system, one processor can spend most of its time
servicing the network interrupts and the other CPU can devote itself to the
actual problem. (This is one of those cases where a multiple processors
may actually show super-linear scalability over single processors for a
specific application.)
Just some stuff to think about. It is never as easy as it sounds. :^)
-James Rogers
jamesr@best.com
This archive was generated by hypermail 2.1.5 : Wed Jul 17 2013 - 04:00:36 MDT