How to setup PXE for Intel OPA (Onmi-Path)

How to setup PXE for Intel OPA (Onmi-Path)

a. Install Intel® Omni-Path Fabric Suite Fabric Manager using the command:
yum install opa-fm
b. Start up Fabric Manager using the following command:
service opafm start

2. The HFI driver (hfi1) and IP over InfiniBand* modules must be running. Use the commands:
modprobe hfi1
modprobe hfi1 modprobe ib_ipoib

5.5.1.1.1 Configure Network Interface
1. Add ib0 network interface configuration file: /etc/sysconfig/network- scripts/ifcfg-ib0
2. Add the following content related to DHCP configuration to the configuration file:

DEVICE=ib0
TYPE=Infiniband
BOOTPROTO=static
IPADDR=192.168.100.1
NETMASK=255.255.255.0
NETWORK=192.168.100.0
BROADCAST=192.168.100.255
ONBOOT=yes
USERCTL=no

3. Run the following command, where ib0 is the name of the network interface configured in step 2:
ifdown ib0
ifup ib0

5.5.1.1.4 Install a TFTP Server
Prerequisite: XINETD may have been installed with OS. If not, use the following command:

yum install xinetd

To install a TFTP server, perform the following steps:
1. Install TFTP package using the command:
yum install tftp-server

2. Add or modify TFTP configuration file. An example is shown below:
etc/ xinetd.d/tftp
service tftp { socket_type = dgram protocol = udp wait = yes user = root server = /usr/sbin/in.tftpd server_args = -s /tftpboot #-s 表示用/tftpboot作为tftp目录的根目录
disable = no #默认是yes，改为no per_source = 11 cps = 100 2 flags = IPv4 }
Note: The disable field defaults to yes and should be changed to no before starting the XINETD server.

3. Start XINETD server using the command:
service xinetd start

4. Enable XINETD service to be run always with OS start:
systemctl enable xinetd.service


5.Configure the FTP Server
#yum install vsftpd
#mkdir /var/ftp/pub/rhel75
copy all files from rhel7.5 iso to /var/ftp/pub/rhel75

5.5.1.1.7 Configure the Boot Loader
To configure a PXE server to use the Grub2 boot loader, perform the following steps.
Note: For configuration details of other boot loaders, please refer to your OS vendor documentation.

1. Copy the shim.efi file from the shim package and the grubx64.efi file from the grub2-efi package in the ISO image file. Enter the following commands as root: (In Packages)
2. Extract the packages.
# rpm2cpio shim-version-architecture.rpm | cpio -dimv
# rpm2cpio grub2-efi-version-architecture.rpm | cpio -dimv

3. Copy the EFI boot images from your boot directory.
# cp publicly_available_directory/boot/efi/EFI/redhat/shim.efi /tftpboot/uefi
# cp publicly_available_directory/boot/efi/EFI/redhat/grubx64.efi /tftpboot/uefi

5.5.2.2 Install RHEL* 7.4 over Intel® OP HFI
To edit the initrd.img, perform the following steps:
1. Log in as root.
2. Decompress the .iso archive with RHEL* 7.4 distribution. The initrd.img is located in the /images/pxeboot folder.
The initrd.img is located in the /images/pxeboot folder.
3. Create /home/images/initrd folder.
mkdir -p /home/images/initrd
4. Copy initrd.img to home/images/initrd.
cp <.iso decompressed archive path>/images/pxeboot/initrd.img /home/images/ initrd
5. Change directories to /home/images/initrd.
cd /home/images/initrd

6. Check archive type. For example:
file initrd.img
initrd.img: XZ compressed data

7. Extract initrd.img according to archive type:
• For XZ archive
xzcat ./initrd.img | cpio -d -i -m
• For GZ archive
gunzip < initrd.img | cpio -i --make-directories

8. Move initrd.img out of the folder.
mv ./initrd.img ../initrd.img

Note: From this point on in the instructions, all commands assume that the present working directory (pwd) is /home/images/initrd and all commands are relative to this path.
9. Copy firmware files.
cp --recursive /lib/firmware/updates/ usr/lib/firmware/updates/
The list of required firmware files are:

/lib/firmware/updates/hfi1_dc8051_d.fw
/lib/firmware/updates/hfi1_dc8051.fw
/lib/firmware/updates/hfi1_fabric_d.fw
/lib/firmware/updates/hfi1_fabric.fw
/lib/firmware/updates/hfi1_pcie_d.fw
/lib/firmware/updates/hfi1_pcie.fw
/lib/firmware/updates/hfi1_sbus_d.fw
/lib/firmware/updates/hfi1_smbus.fw
/lib/firmware/updates/hfi_dc8051.bin

10. Add ib_ipoib.conf file.
mkdir etc/modules-load.d/
touch etc/modules-load.d/ib_ipoib.conf
echo "# Load ib_ipoib.ko at boot" >> etc/modules-load.d/ib_ipoib.conf
echo "ib_ipoib" >> etc/modules-load.d/ib_ipoib.conf

11. Generate initrd-new.img file and give permissions.
cd /home/images/initrd
find . -print |cpio -o -H newc | xz --format=lzma > ../initrd-new.img
chmod 777 ../initrd-new.img

Note: Remember to use appropriate Linux kernel file (vmlinuz) along with initrd.img.

12. Update grub.cfg to include the line insmod=ib_ipoib netdevice=ib0 netsetup=dhcp.
This forces an automatic ib_ipoib module start, uses ib0 as a default network device, and configures IP address via DHCP
#vi /tftpboot/grub.cfg
set default="0"
function load_video {
insmod efi_gop
insmod efi_uga
insmod video_bochs
insmod video_cirrus
insmod all_video
}
load_video
set gfxpayload=keep
insmod net
insmod efinet
insmod tftp
insmod gzio
insmod part_gpt
insmod ext2
set timeout=60

### END /etc/grub.d/00_header ###
search --no-floppy --set=root -l 'CentOS 7 x86_64'
### BEGIN /etc/grub.d/10_linux ###
menuentry 'Install RHEL 7.5 ' --class fedora --class gnu-linux --class gnu --class os {
linuxefi (tftp)/vmlinuz insmod=ib_ipoib netdevice=ib0 netsetup=dhcp inst.repo=ftp://192.168.100.1/pub/rhel
initrdefi (tftp)/initrd-new.img
}

If you PXE fail check PXE server services are on
#service vsftpd status
#service dhcpd status
#service tftp status
#seriver xinetd