MX-10G Software
Downloads and Installation Instructions

Home | Mail for Product Information | Documentation | Software Downloads |
Switch Software | Diagnostic Tools | Other Documentation and Tools |
Technical Support | RMA Procedures

To download Myrinet software, you need a login and password.
If you don't already have these, please request them.

Source | Binary | Myrinet roll for Rocks

MX-10G Software

MX-10G (the Myri-10G version of Myrinet Express) is a low-level message-passing system for 10-Gigabit Myrinet and 10-Gigabit Ethernet networks. The MX software includes the MX library, drivers, Myrinet-NIC control program, and a network mapping program. MX-10G is available for Myri-10G PCI-Express NICs. For Myrinet-2000 PCI-X NICs, refer to the MX-2G Software Download web page.

MX-10G provides kernel-bypass operation for low latency and low host-CPU utilization over either 10-Gigabit Myrinet or 10-Gigabit Ethernet networks. In addition to MX over Myrinet (MXoM) and MX over Ethernet (MXoE), MX-10G includes IP communication (TCP/IP and UDP/IP) over either 10-Gigabit Myrinet (IPoM) or 10-Gigabit Ethernet (IPoE).

For application or middleware developers who need to develop code using the MX API, refer to the MX API manual. MX-10G for Myri-10G PCI-Express NICs and MX-2G for Myrinet-2000 PCI-X NICs are fully compatible at the API and application levels, and use the same MPI and Sockets middleware.

Complete installation instructions and platform-specific information can be found in the README in the top-level directory of the source distribution.

Source code is available for all distributions. However,

• WHQL-certified drivers and MSI installers for Windows XP and 2003 are available, and will be available soon for Windows Vista and Windows Server 2008 (Longhorn).
• MX-10G software for Linux, Solaris, and Mac OS X is distributed only in source form.
• An MX-10G roll for Rocks v5.1 is available.

* Note for recent Linux versions: we try to support newer Linux versions within a few weeks of their official release. We also encourage customers to report any problem encountered with development Linux versions or release candidates.

For 10G-PCIE-8B-* NIC support, MX 1.2.8 or later is required.

Note:

• To obtain diagnostic information about the operation of the Myri-10G NICs download the 10G NIC Tool Kit.
• Xen virtualization software is supported.
• There are currently some restrictions for MXoE and MXoM on two different NICs at the same time on the same machine. While it is supported, there are some manual configuration steps necessary. Contact help@myri.com for details.
• The MXoM distribution can be used in a point-to-point (switchless) configuration or with the NICs connected to a Myri-10G switch.
• The MXoE distribution can be used in a point-to-point (switchless) configuration or with the NICs connected to a low-latency 10-Gigabit Ethernet switch. The 10-Gigabit Ethernet switch must support jumbo frames, and flow control should be enabled on the switch.
• MX is supported on the following Linux distributions: RHEL kernels ≥ RHEL3, SuSE kernels ≥ SuSE-8.0, and any kernel based on a kernel.org version ≥ 2.4.23. We provide support for other kernels on a per-request basis. Contact help@myri.com for details.
• If you would like to compile MX on AMD64 running a Solaris 10 release after s10u1_18, you must apply patches 118344, and 118844. For patch access, refer to the SunSolve Patch Access web page.

Here is a summary of the CHANGES between MX versions.

Abbreviated Installation Instructions

To compile MX on Linux platforms, you will need GNU make and C/C++ compilers. Please be sure to read the README file for detailed installation instructions and architecture-specific troubleshooting. Unresolved questions may also be addressed in the FAQ.

Customer compilation of MX for Windows is not recommended due to the difficulties involved. Users should be comfortable with the Cygwin environment. Contact help@myri.com for instructions.

Current cautions and common problems:

• MX must be compiled with the system compiler (gcc for Linux, Solaris/AMD64, and Mac OS X). For Solaris/UltraSPARC, MX must be compiled with the SunPRO compiler. We do not support third-party compilers.
• If you would like to compile MX on AMD64 running a Solaris 10 release after s10u1_18, you must apply patches 118344, and 118844. For patch access, refer to the SunSolve Patch Access web page.
• If you are installing MX on Linux, you must configure/compile/load MX on a Linux box whose running kernel is configured to match the source kernel tree. Note that some Linux distributions (e.g., SuSE, RedHat, and Fedora) ship a mismatched source kernel tree.
• For optimal performance, write-combining must be enabled on the PCI chipset of the host. The MX driver will automatically attempt to enable write-combining. If MX is unable to enable write-combining at load time, an error message will appear in the kernel log. Refer to the README for details.
• For application or middleware developers who need to develop code using the MX API, refer to the MX API manual.
• If the host is rebooted, you must reload the MX drivers.

MX installation is performed in four steps:

1. Configuring and compiling MX.

	     $ gunzip -c mx_1.2.9.tar.gz | tar xvf -
	     $ cd mx-1.2.9
	     $ ./configure
	     $ make

By default, we assume that you would like to use MX in Myrinet mode. If you are using MX-10G, and you would like to use MX in Ethernet mode, you must configure MX-10G with the following option:

	     $ configure --enable-ether-mode
    

By default, we assume that the header and config files of your Linux kernel (required to compile outside modules and either part of a kernel-headers or kernel-source package depending on your distribution) are pointed by /lib/modules/`uname -r`/{source,build}. If your Linux installation is not standard, or you are cross-compiling for a kernel different from the one on the compile node, refer to the README for configure instructions.

Note:

• Additional configure options are listed in the README.

• By default, the mapper in MXoM is provided by the Fabric Management System (FMS). Optionally, if you would like to use the diagnostic capabilities of FMS on MXoM, you need to specify the name of the FMS server (the node on which the fms process will be run) at configure time, using --with-fms-server=<fms_server>.

  	     $ ./configure --with-fms-server=<fms_server>
  	

  To defer this specification until install time, or to override it, you may install MXoM with make install FMS_SERVER=<fms_server>.

  For detailed installation instructions for MXoM with FMS diagnostic monitoring, refer to the FMS webpage.

2. Installing the MX mcp and driver.

Select an installation directory path <install_path>. It is usually best for <install_path> to be the path to an NFS directory available on all machines that are to share this MX installation. The directory must be accessible using <install_path> on all machines that are to share the installation. <install_path> must be an absolute path; it must start with /. However, <install_path> may contain symbolic links.

	     $ make install prefix=<install_path>

If you omit prefix=<install_path>, the mcp and driver will be installed in the directory specified with the configure --prefix option, or the default directory, /opt/mx/. The MX binaries are located in <install_path>/bin and <install_path>/sbin.

For Linux, add the MX library directory to the system library search path. Otherwise, individual users will have to either manage their LD_LIBRARY_PATH environment variable or link their program with an -rpath option for the dynamic linker to locate the MX shared library.

Next, you must run

	     $ su root
	     $ <install_path>/sbin/mx_local_install
	     $ <install_path>/sbin/mx_start_stop start
	  

on each machine to perform local install steps, to load the modules, and to start a mapper for each Myrinet NIC contained in the machine. If applicable, the mx_start_stop script is also available in /etc/init.d/mx.

Important notes:

• Details of the installation scripts can be found in the README.
• The MX software provides a separate kernel module for the mcp (firmware) and the driver. If you do not use the mx_start_stop script to load the MX drivers and start the MX mapper, you must ensure that the mcp module is loaded first, as the driver module depends on it. The MX driver and mcp contain a number of tunable parameters which may be adjusted by the user when the MX driver is loaded. These parameters are set in an OS-dependent manner. Refer to the README for details.
• For Linux, add the MX library directory to the system library search path. Otherwise, individual users will have to either manage the LD_LIBRARY_PATH environment variable or link the program with an -rpath option for the dynamic linker to locate the MX shared library.
• The 32-bit MX libraries are installed in <install_path>/lib32 and the 64-bit MX libraries are installed in <install_path>/lib64. The <install_path>/lib directory is a symbolic link to either lib64 or lib32 depending on the native wordsize detected by configure. E.g., on most ppc64 distributions, gcc defaults to 32-bit, which means that lib links to lib32. However, on most x86_64 distributions, gcc defaults to 64-bit, so lib links to lib64.
• Unless specified on the configure line, MX builds 32-bit libraries on 32-bit architectures (i386, ppc) and 64-bit libraries on 64-bit architectures (ia64, AMD64, ppc64). It is possible to build both by using the --enable-32b and --enable-64b configure flags.
• For Mac OS X, when the Apple Xcode compiler is 64-bit, MX fat libraries are built which are usable by both 32-bit and 64-bit applications, and libraries are always installed in <install_path>/lib.
• The installation scripts do not configure the IP device. If you wish to run IP over MX/Myrinet (ethernet emulation), you must configure the device. Refer to step 3.

3. Enabling IPoM or IPoE (OPTIONAL)

Instructions for Linux:

If you wish to run IP communication (TCP/IP and UDP/IP) over either 10-Gigabit Myrinet (IPoM) or 10-Gigabit Ethernet (IPoE), the command to enable IPoM or IPoE is as follows:

	   /sbin/ifconfig myri0 <ip_address> up

where you must replace myri0 with the appropriate name (myri1, myri2, etc.) if you have more than one Myrinet NIC per host.

Instructions for Solaris:

If you wish to run IPoM, the command to enable IPoM is as follows:

	   ifconfig myri0 plumb <ip_address> up

where you must replace myri0 with the appropriate name (myri1, myri2, etc.) if you have more than one Myrinet NIC per host.

• Note for Solaris 10GA: Due to a bug, Solaris 10GA does not support 9000 byte (jumbo) frames as released. In order to obtain full ethernet performance, jumbo frames are critical, so we encourage you to apply Sun's patch 119832-01 for UltraSPARC or patch 119833-01 for AMD64. For patch access, refer to the SunSolve Patch Access web page. If you do not apply this patch, you will need to set the Ethernet MTU to 1500 bytes in /kernel/drv/myri.conf, and specify mx_mtu_override=1500 when loading MX. Refer to the README for details. No jumbo frames will be allocated until after the myri0 interface is plumbed.

Instructions for Mac OS X:

You should configure the MX ethernet emulation interface as you would any other ethernet interface. On most systems, the MX ethernet adaptor will appear as en1. It is possible, if you have additional network cards, that the adaptor will appear as en2, en3, etc.

To verify which ethernet adaptor belongs to MX, you may need to run the Network Utility /Applications/Utilities/Network Utility. Click the Info tab and select Network Interface from the menu until you find the one whose Vendor is Myricom, and whose Hardware Address matches the MAC address printed by mx_info.

Once you have found the correct adaptor, configure it via:

	  System Preferences -> Network -> Show -> Ethernet Adaptor (enX)
	  

4. Testing the MXoM/MXoE Installation.

Once the MXoM firmware is running on all of the hosts, and all host-to-switch and switch-to-switch cables have been connected, you should verify that all hosts are mapped in the network (via the mx_info command) before launching jobs on the network. We also recommend that you check the health of all of the Myricom hardware components in the Myrinet installation as instructed in the Troubleshooting section of the FAQ.

Once the MXoE firmware is running on all of the hosts, and all host-to-switch cables have been connected, all hosts should become rapidly visible in the output of mx_info -a.

Congratulations! You have successfully installed MX.

Refer to the Software and Documentation Page for details of MX-based middleware (MPICH-MX, Sockets-MX). If you encountered difficulties during any phase of this installation process, we recommend that you first consult the README and FAQ, and if you still have unresolved questions, please feel free to contact technical support at help@myri.com.

Binary Installation Instructions

The binary Windows driver can be installed on the command line or via the graphical interface. The instructions for installing the MXoM or MXoE driver on the command line can be found in the respective README for MXoM or README for MXoE. The user interface for installing a device has changed with each new release of Windows. Steps for each OS are given below. We suggest booting into safe mode and uninstalling old drivers before reinstalling new ones.

• Windows 2000 installation instructions
• Windows XP installation instructions

Note: Each Myrinet roll contains MX-10G 1.2.8, MPICH2-MX 1.0.7..2, MPICH-MX 1.2.7..7, and Open MPI v1.2.8. Installation instructions are available on the Myrinet FAQ and the Rocks homepage.

Last updated: 24 June 2009

Home | Mail for Product Information | Documentation | Software Downloads |
Switch Software | Diagnostic Tools | Other Documentation and Tools |
Technical Support | RMA Procedures