Myricom at Baltimore, Maryland 18-21 November 2002

The Exhibit Team. The Myricom team attending and exhibiting at SC2002 had a great time. The Myricom booth was crowded most of the time (the photo above was taken one morning before the exhibits opened), and the overall attendance appeared to be up compared with earlier SC conferences. We saw many old friends, and met some new ones. The Myricom attendees were Susan Blackford, Member Technical Staff; Nan Boden, Executive Vice President; Glenn Brown, Senior Programmer; Nelson Escobar, Programmer; Reese Faucette, Senior Programmer; Andrew Gallatin, Senior Programmer; Patrick Geoffray, Senior Programmer; Gay Lynn Moore, Director of Public Relations; Dave PeGan, Vice President, Sales; Chuck Seitz, CEO & CTO; Jakov Seizovic, Member of the Technical Staff; and Ruth Siviolotti, Member of the Technical Staff.

Other Highlights. In addition to the exhibits, the list of TOP500 supercomputer sites was released Friday 15 November before SC2002. Myrinet made another excellent showing with 140 of the TOP500 systems using Myrinet. Please see this separate Nov-2002 TOP500 news item for the details. On Wednesday afternoon, Chuck Seitz, Myricom's CEO & CTO, gave a very well attended and received Masterworks talk titled "The Architecture, Programming, and Applications of Clusters." On Wednesday evening, there was a Myrinet User's Group (MUG) Birds of a Feather (BOF) session. Another memory of SC2002 was the Myrinet penguins, nearly 800 of them given away as "favors."

Booth Demonstrations. The demonstrations in the Myricom booth included:

• NAMD molecular-modeling application running on a dual-Xeon cluster.
• Mute monitoring tools running on an Itanium-2, and other Itanium-2 demos on HP zx2600 and zx6000 hosts.
• Myrinet-Ethernet interoperability, including netperf TCP/IP benchmarks of 1.95 Gbits/s between Xeon hosts running GM 2.0, and new switch line cards with GbE ports. The interoperability of the GbE ports on these line cards was easily demonstrated using a laptop with a GbE port. See the story below for details.

Please plan to visit Myricom at SC2003 in Phoenix!

Myricom features Myrinet-Ethernet Interoperability in SC2002 demonstrations

People who already use Myrinet™ know that Myrinet carries ethernet packets even more efficiently than Gigabit ethernet. The standard netperf benchmarks on fast hosts under Linux or FreeBSD show one-way TCP/IP and UDP/IP data rates approaching the Myrinet channel rate of 2 Gbits/s. See this web page for the performance details, including the message latency and host-CPU utilization.

How does Myrinet "ethernet emulation" work? Myrinet allows multiple protocols to co-exist on a Myrinet network by tagging each packet with a packet type. The standard GM message-passing system has its own Myrinet types for OS-bypass communication, which exhibits lower host-CPU overhead and lower latency than ethernet emulation. In addition to its special OS-bypass functions, the GM device driver also presents itself to the host operating system as an ethernet device driver, and emulates ethernet by encapsulating ethernet packets as Myrinet packets whose type is "ethernet." GM routes these packets according to Myrinet's map of host IP or ethernet-MAC addresses. Thus, Myrinet carries any packet traffic and protocols that can be carried on ethernet, including TCP/IP and UDP/IP.

How do Myrinet customers use ethernet emulation? For technical-computing sites, it is common that MPI (MPICH-GM over GM) OS-bypass traffic for HPC tasks is mixed with TCP/IP traffic for file operations. For commercial applications, such as distributed databases written to operate over ethernet, Myrinet offers higher data rate and lower latency communication than native ethernet, while still allowing ethernet channel bonding for high-availability (no single point of failure) applications.

One of Myricom's demonstrations at SC2002 was of the performance of Myrinet ethernet emulation. TCP/IP data rates measured by the netperf benchmark were 1.95 Gbits/s between a pair of dual 2.4GHz Xeon hosts under Linux 2.4.19 and GM 2.0. The host-CPU utilization at 1.95 Gbits/s was ~15%. However, this CPU load was on a dual Xeon with hyperthreading enabled, so it amounts to ~60% of one execution thread. These GM 2 results are slightly better than GM 1 ethernet emulation because of GM 2 optimizations such as interrupt coalescing.

Ethernet emulation has been a part of Myricom's Myrinet software since Myrinet was introduced in 1994. What's new is:

• Faster hosts, and more efficient OS IP protocol stacks, fast enough to allow IP traffic to reach similar data rates as OS-bypass traffic.
• The ethernet-emulation code itself has improved.
• Myricom has developed special Myrinet-switch line cards with the physical-level and data-link-level conversion between ethernet/Myrinet and native Gigabit ethernet built into switch ports. This conversion is accomplished with a Lanai XM chip -- the first in the family of Lanai X (10) chips -- on each of the GbE ports. The RISC in the Lanai XM executes the protocol-conversion firmware.

Myrinet-ethernet interoperability was formerly accomplished by installing a Myrinet interface and an ethernet interface in a host with IP forwarding enabled. The special Myrinet switch line cards with GbE ports make interoperability more convenient, lower in cost, and higher in performance. In fact, you can look ahead to Myrinet switches -- the ultimate in scalability -- with hundreds or thousands of GbE ports for low-cost GbE "Beowulf" clusters.

Programmable ports on Myrinet-switch line cards have opened Pandora's box for interoperability between Myrinet (inside) switches and almost any protocol (outside). In addition to GbE over either copper or fiber, the Lanai XM chip can support long-range-Myrinet protocols (up to 100s of km).

22 November 2002