Download latest OFED SW
https://www.openfabrics.org/downloads/OFED/
BIOS setting of SUT:
v Disable SpeedStep/Hyper-threading/C3/C6 in BIOS CPU configuration.
The test steps are seperated to
1.Install OFED,
2. ib IP setting
3.ssh key setting
4.run script
---------------------------------------------------------------------------
1.Install OFED
#cd OFED
#./install.sh
From the following menu select 2 “Install OFED Software”
InfiniBand OFED Distribution Software Installation Menu
1) View OFED Installation Guide
2) Install OFED Software
3) Show Installed Software
4) Configure IPoIB Network Interface and OpenSM Server
5) Uninstall InfiniBand Software
6) Build OFED Software RPMs
Q) Exit
Select Option [1-6]:
|
From the following sub-menu select 3 “All packages (all of openib, mpi)”
Select OFED Software
1) Basic (InfiniBand modules and basic user level libraries)
2) HPC (InfiniBand modules and libraries, mpi and diagnostic tools)
3) All packages (all of openib, mpi)
4) Customize
Q) Exit
Select Option [1-4]:
|
After selecting the complete installation you will be prompted for the compiler used for compiling the 2 MPI packages included in the distribution. Accept the default of gcc when prompted.
Install complete and successfully.
2. ib IP setting
#/etc/init.d/openibd start
#service opensmd start#chkconfig opensmd on
make sure state is Active in ibstat
#ibstat
Set ib ip address in SUT1
#ifconfig ib0 10.0.0.1 255.255.255.0
Set ib ip address in SUT2
#ifconfig ib0 10.0.0.2 255.255.255.0
Disable iptables on both SUT1 and SUT2
#service iptables stop
Check SUT1 and SUT2 connect
(SUT1)# ping 10.0.0.2
3.ssh key setting
From the host 10.0.0.1 perform the following steps:
#ssh 10.0.0.2 (ssh “ip”)
At the prompt similar to the following, type yes and press Enter.
----------------------------------------------------------------------------------------------------photo
The authenticity of host 'localhost (10.0.0.2)' can't be established.
RSA key fingerprint is 23:35:92:2b:df:5b:d6:1b:6a:32:24:ee:18:d2:82:18.
Are you sure you want to continue connecting (yes/no)?
-----------------------------------------------------------------------------------------------------
Enter root password then type exit.
Generate authentication keys
cd ~/.ssh
ssh-keygen –t rsa
Continue to press enter to accept default file name and no passphrase.
-----------------------------------------------------------------------------------------------------------photo
[root@localhost ~/.ssh]# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
7a:e0:dc:a5:ec:99:0a:7a:33:4f:d6:ee:9d:5c:7d:c2 root@localhost
------------------------------------------------------------------------------------------------------------
Create the authorized_keys file for uutn1.
cat id_rsa.pub >> authorized_keys
Copy the authorized_keys file to the initiator host uutn1.
scp authorized_keys 10.0.0.2:~/.ssh/10.0.0.2.id
From the initiator host 10.0.0.2 perform the following steps.
cd ~/.ssh
cat 10.0.0.2.id >> authorized_keys
Re-do test, you have to clear the old key
Clear all existing auth-keys in SUT1 (unsvr) and SUT2 (unsvr1)
SUT1) # rm -rf ~/.ssh/*
SUT2) # rm -rf ~/.ssh/*
SUT1) # rm -rf ~/.ssh/*
SUT2) # rm -rf ~/.ssh/*
4.run script
install perftest-2.0-2.el6.x86_64.rpm
(Download from :http://mirror.centos.org/centos/6/os/x86_64/Packages/perftest-2.0-2.el6.x86_64.rpm)
#rpm -ivh perftest-2.0-2.el6.x86_64.rpm--force
1Open a new file using an editor such as vi and copy the sample script from below into the file. Save the file and use chmod to make it executable
#vim runall
chmod +x ./runall
#!/bin/sh
# trivial script to launch all server/client bandwidth and latency
# tests with ssh. Must be launched from client. Uses test defaults
# and runs both unidirectional and bidirectional bandwidth tests.
# examples: runall <servername> or runall <server_ip>
BW_TESTS="rdma_bw ib_read_bw ib_send_bw ib_write_bw"
LAT_TESTS="rdma_lat ib_read_lat ib_send_lat ib_write_lat"
OFED_BINDIR=/usr/bin
SERVER=$1
run ()
{
server=$1
shift
ssh $server $* >& /dev/null &
#give server time to start
sleep 2
$* $server
status=$?
wait
return $status
}
if [ $# -lt 1 ] ; then
echo "Usage: runall <server>"
exit 3
fi
for test in $BW_TESTS ; do
run $SERVER $OFED_BINDIR/$test
run $SERVER $OFED_BINDIR/$test -b
done
for test in $LAT_TESTS ; do
run $SERVER $OFED_BINDIR/$test
done
|
2. Run the script. If desired, you can redirect the output to a file for inclusion in the final report. A sample of the output is provided below.
./runall <server name or IP address>
Or
./runall <server name or IP address> > <output file>
local address: LID 0x05, QPN 0x830405, PSN 0xbc893c RKey 0x7a002400 VAddr 0x002aedf208f000
remote address: LID 0x04, QPN 0x810405, PSN 0xe022f6, RKey 0x12002400 VAddr 0x002b1b7d7fb000
Bandwidth peak (#0 to #989): 744.555 MB/sec
Bandwidth average: 744.527 MB/sec
Service Demand peak (#0 to #989): 4535 cycles/KB
Service Demand Avg : 4535 cycles/KB
local address: LID 0x05, QPN 0x840405, PSN 0xaf75dc RKey 0x002400 VAddr 0x002b5970ce6000
remote address: LID 0x04, QPN 0x820405, PSN 0xca1275, RKey 0x18002400 VAddr 0x002af22122a000
Bandwidth peak (#0 to #890): 1344.07 MB/sec
Bandwidth average: 1343.9 MB/sec
Service Demand peak (#0 to #890): 2512 cycles/KB
Service Demand Avg : 2512 cycles/KB
------------------------------------------------------------------
RDMA_Read BW Test
Connection type : RC
local address: LID 0x05, QPN 0x850405, PSN 0x416402 RKey 0x6002400 VAddr 0x002ab8a3128000
remote address: LID 0x04, QPN 0x830405, PSN 0x1d2d1e, RKey 0x1e002400 VAddr 0x002ad71ee91000
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
65536 1000 722.06 722.04
------------------------------------------------------------------
------------------------------------------------------------------
RDMA_Read Bidirectional BW Test
Connection type : RC
local address: LID 0x05, QPN 0x860405, PSN 0xb27762 RKey 0xc002400 VAddr 0x002b353f60e000
remote address: LID 0x04, QPN 0x840405, PSN 0xf4a018, RKey 0x24002400 VAddr 0x002b42dc3e2000
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
65536 1000 1213.91 1213.91
------------------------------------------------------------------
------------------------------------------------------------------
Send BW Test
Inline data is used up to 400 bytes message
Connection type : RC
local address: LID 0x05, QPN 0x870405, PSN 0x893f8f
remote address: LID 0x04, QPN 0x850405, PSN 0x52149e
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
65536 1000 743.12 743.12
------------------------------------------------------------------
------------------------------------------------------------------
Send Bidirectional BW Test
Inline data is used up to 400 bytes message
Connection type : RC
local address: LID 0x05, QPN 0x880405, PSN 0x7879ae
remote address: LID 0x04, QPN 0x860405, PSN 0xbd3577
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
65536 1000 1319.16 1319.16
------------------------------------------------------------------
------------------------------------------------------------------
RDMA_Write BW Test
Inline data is used up to 400 bytes message
Number of qp's running 1
Connection type : RC
Each Qp will post up to 100 messages each time
local address: LID 0x05, QPN 0x890405, PSN 0xf51a5b RKey 0x1e002400 VAddr 0x002ab514275000
remote address: LID 0x04, QPN 0x870405, PSN 0x82d78, RKey 0x36002400 VAddr 0x002b8f7954e000
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
65536 5000 744.72 744.72
------------------------------------------------------------------
------------------------------------------------------------------
RDMA_Write Bidirectional BW Test
Inline data is used up to 400 bytes message
Number of qp's running 1
Connection type : RC
Each Qp will post up to 100 messages each time
local address: LID 0x05, QPN 0x8a0405, PSN 0x17cc0 RKey 0x24002400 VAddr 0x002af1b8b05000
remote address: LID 0x04, QPN 0x880405, PSN 0x23d292, RKey 0x3c002400 VAddr 0x002af1a056b000
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations BW peak[MB/sec] BW average[MB/sec]
65536 5000 1345.45 1345.45
------------------------------------------------------------------
local address: LID 0x05 QPN 0x8b0405 PSN 0x61f8fa RKey 0x2a002400 VAddr 0x00000000509001
remote address: LID 0x04 QPN 0x890405 PSN 0x8879fe RKey 0x42002400 VAddr 0x00000000509001
Latency typical: 3.8045 usec
Latency best : 3.72367 usec
Latency worst : 179.276 usec
------------------------------------------------------------------
Read Req Latency Test
local address: LID 0x05 QPN 0x8c0405 PSN 0x168f09 RKey 0x30002400 VAddr 0x0000000050c080
remote address: LID 0x04 QPN 0x8a0405 PSN 0x287769 RKey 0x48002400 VAddr 0x0000000050c080
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec]
2 1000 8.49 18.40 8.61
------------------------------------------------------------------
------------------------------------------------------------------
Send Latency Test
Inline data is used up to 400 bytes message
Connection type : RC
local address: LID 0x05 QPN 0x8d0405 PSN 0xf2d6d8
remote address: LID 0x04 QPN 0x8b0405 PSN 0x3da316
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec]
2 1000 5.08 63.87 5.13
------------------------------------------------------------------
------------------------------------------------------------------
RDMA_Write Latency Test
Inline data is used up to 400 bytes message
Connection type : RC
local address: LID 0x05 QPN 0x8e0405 PSN 0x5c8536 RKey 0x3e002400 VAddr 0x0000000050c002
remote address: LID 0x04 QPN 0x8c0405 PSN 0xcb23fc RKey 0x56002400 VAddr 0x0000000050c002
Mtu : 2048
------------------------------------------------------------------
#bytes #iterations t_min[usec] t_max[usec] t_typical[usec]
2 1000 3.70 24.04 3.79
------------------------------------------------------------------
|
