Myricom, the company that pioneered high-performance cluster interconnect, is now shipping Myri-10G products, a convergence that leverages 10-Gigabit Ethernet technology into the HPC world, and HPC techniques into the Ethernet world.

The new Myri-10G solutions are 10-Gigabit Ethernet from Myricom, and more. Myri-10G’s Myrinet extensions open a broader universe of system solutions than Ethernet alone can provide.

In Ethernet mode with the bundled drivers for Linux, Windows, Solaris, Mac OS X, or FreeBSD, Myri-10G network-interface cards (NICs) deliver maximal performance at minimal cost, and are fully compliant with Ethernet standards. Connect the NIC port to your 10-Gigabit Ethernet switch and you’ll see near-wire-rate TCP/IP and UDP/IP throughput (9.7-9.9 Gbits/s).

The broader software support for 10-Gigabit Myrinet and Low-Latency 10-Gigabit Ethernet is MX (Myrinet Express). Connect the NIC to a 10-Gigabit Myrinet switch or to a low-latency 10-Gigabit Ethernet switch, and MX unleashes the capabilities of a network that supports the same software interfaces as Ethernet, plus low-latency, kernel-bypass communication using MPI and Sockets APIs. Also, 10-Gigabit Myrinet is interoperable with 10-Gigabit Ethernet for connections to IP networks (grids) and storage.

Myri-10G dual-protocol NICs

Myri-10G NICs connect to hosts through PCI-Express x8, a 2+2 GigaByte/s full-duplex IO fabric that is fast enough to keep up with the 1.25+1.25 GigaByte/s network port. A PCI-X bus, 1 GigaByte/s at best, has only 40% of the 2.5 GigaByte/s peak data rate needed for 10-Gigabit networking. Myri-10G NICs are the first near-wire-speed 10-Gigabit NICs.

These NICs, like all earlier Myrinet NICs, include processors and firmware to offload network protocol processing, minimizing host-CPU utilization and enabling low-latency kernel-bypass communication directly with applications.

• Physical Links are 10-Gigabit Ethernet
– At the Physical level (PHY, layer-1), the ports of Myricom's custom Myri-10G chips are XAUI, per IEEE 802.3ae. The data rate is 10+10 Gigabits per second, full-duplex.
– XAUI is readily converted to other 10-Gigabit Ethernet PHYs. The ports of current NIC and switch products are 10GBase-CX4, 10GBase-R, or XAUI over ribbon fiber. See this quick reference on Myri-10G PHYs (pdf, 35KB).
– At the Data-Link level (layer-2), the links may operate with either Ethernet or Myrinet protocols.

• Industry-standard Cables: choice of copper or fiber
– 10GBase-CX4 copper cables to 15m or ribbon-fiber cables to 200m are the best choices for low-latency applications.
– 10GBase-R serial fiber is available for 10-Gigabit Ethernet applications.

• NICs are dual-protocol
– These PCI-Express x8 NICs are firmware-controlled offload engines, just like all earlier Myrinet NICs.
– With the bundled 10-Gigabit Ethernet driver and firmware, the NIC operates as a protocol-offload 10-Gigabit Ethernet NIC.
– With the optional MX (Myrinet Express) driver and firmware, the NIC supports the same IP communication as with the Ethernet software, plus low-latency, kernel-bypass operation over either 10-Gigabit Myrinet or 10-Gigabit Ethernet networks.

• Switches are 10-Gigabit Myrinet Internally
– Although Myri-10G leverages 10-Gigabit Ethernet technology at the link level, it retains the efficiency, simplicity, and scalability of Myrinet switching.
– These 10-Gigabit Myrinet switches and full-bisection Clos “Network in a box” products are similar except for data rates to Myrinet-2000 switches.
– Special switch line cards provide a mix of Ethernet and Myrinet external ports. The internal switching is Myrinet, with protocol conversion to external Ethernet ports.

• Performance
– MX kernel-bypass mode with Intel MPI benchmarks: 2.3µs MPI latency with MPICH-MX, 1.2 GBytes/s one-way (PingPong) data rate, and 2.4 GBytes/s two-way (SendRecv) data rate with a Myrinet switch. With a low-latency 10-Gigabit Ethernet switch, the data rates are the same, but the latency is in the 2.6-2.8µs range
– 10-Gigabit Ethernet mode, or under MX-10G ethernet emulation: 9.6 Gbits/s TCP/IP rate (netperf benchmarks with Linux 2.6.9, jumbo frames, default offload parameters).

25 October 2006