vSphere 5 Network PXE install

Note to my regular readers: There is no cycling or transportation content in this article about installing and booting VMware ESXi 5.x via the network, which is related to my day job of operating systems bringup in a group that designs new x86 server hardware.


VMware vSphere ESXI 5.5 PXE Install

Assumptions

This article assumes some knowledge of datacenter basics and how PXE booting works in a mostly Linux environment. To summarize: The “Preboot eXecution Environment” (PXE, pronounced “pixie”) for computer network interfaces provides a very quick and convenient way to boot and install operating system software in an enterprise environment after the initial setup of your PXE environment. Using PXE does require some infrastructure to support it. You need a system to serve up DHCP addresses. In a switched environment, your network should know how to forward DHCP requests to the correct server. Your DHCP server provides information about the location of PXE server.

I also assume you know how to extract files from an ISO install image, and that you’ve done so with the VMware vSphere 5.x server install ISO. You should see several files and a couple of subdirectories that look something like this:

a.b00         boot.cfg      ipmi_ipm.v02  misc_dri.v00  net_mlx4.v00  sata_sat.v01  scsi_ips.v00  tools.t00
ata_pata.v00  chardevs.b00  isolinux.bin  mtip32xx.v00  net_mlx4.v01  sata_sat.v02  scsi_lpf.v00  uhci_usb.v00
ata_pata.v01  efi           isolinux.cfg  net_be2n.v00  net_nx_n.v00  sata_sat.v03  scsi_meg.v00  upgrade
ata_pata.v02  efiboot.img   jumpstrt.gz   net_bnx2.v00  net_tg3.v00   sata_sat.v04  scsi_meg.v01  user.b00
ata_pata.v03  ehci_ehc.v00  k.b00         net_bnx2.v01  net_vmxn.v00  sb.v00        scsi_meg.v02  useropts.gz
ata_pata.v04  elxnet.v00    ks.cfg        net_cnic.v00  ohci_usb.v00  scsi_aac.v00  scsi_mpt.v00  vmware-esx-base-osl.txt
ata_pata.v05  esx_dvfi.v00  lpfc.v00      net_e100.v00  qlnative.v00  scsi_adp.v00  scsi_mpt.v01  vmware-esx-base-readme
ata_pata.v06  ima_qla4.v00  lsi_mr3.v00   net_e100.v01  rste.v00      scsi_aic.v00  scsi_mpt.v02  weaselin.t00
ata_pata.v07  imgdb.tgz     lsi_msgp.v00  net_enic.v00  safeboot.c32  scsi_bnx.v00  scsi_qla.v00  xlibs.v00
b.b00         imgpayld.tgz  mboot.c32     net_forc.v00  sata_ahc.v00  scsi_bnx.v01  scsi_qla.v01  xorg.v00
block_cc.v00  ipmi_ipm.v00  menu.c32      net_igb.v00   sata_ata.v00  scsi_fni.v00  s.v00
boot.cat      ipmi_ipm.v01  misc_cni.v00  net_ixgb.v00  sata_sat.v00  scsi_hps.v00  tboot.b00

For now, I discuss PXE install and network boot for legacy BIOS using SYSLINUX (not gPXE). VMware doesn’t seem to support network install for UEFI systems, though I’m working on that too. These instructions work for VMware ESXi 5.0 through 5.5 Update 2.

I assume you have read the VMware documents for network install. Read and try VMware’s suggested process first. VMware’s instructions work as long as you’re at a homogenous VMware lab environment. Some of you got lucky and just happen to use the same version of SYSLINUX used by VMware when they compiled their custom boot loader, or you built your PXE environment from scratch. The rest of you all, come back here when you run into trouble and need to troubleshoot.

SYSLINUX

I work in a SYSLINUX environment, so I’ll describe what I found here.

SYSLINUX is H. Peter Avin’s collection of open source boot loaders. Several binaries are created using a common set of header files, and these binaries interact with each other.

VMware uses version 3.86 of SYSLINUX to compile their boot loaders for all revisions of ESXi 5.x (as of January 2015). The interesting binaries for this version of SYSLINUX are pxelinux.0, menu.c32, and mboot.c32.

Like most OS vendors, VMware includes their own mboot.c32 binary on their install ISO image. Some of you already understand you need to use the mboot.c32 that comes with your version of SYSLINUX, and can’t figure out why your VMware PXE install doesn’t work.

That’s because VMware hosed you — their mboot.c32 is not the SYSLINUX mboot.c32! Other than the options for mboot.c32 mentioned in VMware’s docs that don’t belong with SYSLINUX, this is documented nowhere unless you happened to download VMware’s open source for vSphere 5.x and found the code for VMware’s multiboot loader at server/esxboot-1.00/mboot/.

VMware’s mboot is compiled using SYSLINUX 3.86 header files, so you must have pxelinux.0 and menu.c32 binaries compiled from the version 3.86 source code of SYSLINUX. You can find that source in the VMware open source download linked above, and find it at server/syslinux-3/syslinux-3.86/. Compile on any reasonably recent 64-bit Linux with a development environment.

If you need the real SYSLINUX mboot.c32 to, for example, install Xen, you can rename VMware’s mboot.c32. In my environment, I call it vboot.c32. Your pxelinux menu file at pxelinux.cfg/default should look something like this now. Note the KERNEL line.

DEFAULT menu.c32
MENU TITLE ESXi 5.X

LABEL ESXi 5.5.2 Manual install
  KERNEL vboot.c32
   APPEND -c boot.cfg
MENU LABEL ESXi 5.5.2 ^Manual install

Understanding that VMware’s mboot.c32 is not the same as the SYSLINUX mboot.c32 should solve most of your problems. In the discussion below, I’ll call VMware’s boot loader “vboot.c32” to cut down on the confusion.

boot.cfg

If you run vboot.c32 without any options (i.e. you remove the “APPEND” line in the pxelinux menu), it will load the file boot.cfg by default. You can find a default boot.cfg in VMware’s install ISO. This default boot.cfg initiates the usual manual install process.


bootstate=0
title=Load ESX Installer
kernel=tboot.b00
kernelopt=runweasel
modules=b.b00 --- jumpstrt.gz --- useropts.gz --- k.b00 --- chardevs.b00 --- a.b00 --- user.b00 --- sb.v00 --- s.v00 --- ata_pata.v00 --- ata_pata.v01 --- ata_pata.v02 --- ata_pata.v03 --- ata_pata.v04 --- ata_pata.v05 --- ata_pata.v06 --- ata_pata.v07 --- block_cc.v00 --- ehci_ehc.v00 --- elxnet.v00 --- weaselin.t00 --- esx_dvfi.v00 --- xlibs.v00 --- ima_qla4.v00 --- ipmi_ipm.v00 --- ipmi_ipm.v01 --- ipmi_ipm.v02 --- lpfc.v00 --- lsi_mr3.v00 --- lsi_msgp.v00 --- misc_cni.v00 --- misc_dri.v00 --- mtip32xx.v00 --- net_be2n.v00 --- net_bnx2.v00 --- net_bnx2.v01 --- net_cnic.v00 --- net_e100.v00 --- net_e100.v01 --- net_enic.v00 --- net_forc.v00 --- net_igb.v00 --- net_ixgb.v00 --- net_mlx4.v00 --- net_mlx4.v01 --- net_nx_n.v00 --- net_tg3.v00 --- net_vmxn.v00 --- ohci_usb.v00 --- qlnative.v00 --- rste.v00 --- sata_ahc.v00 --- sata_ata.v00 --- sata_sat.v00 --- sata_sat.v01 --- sata_sat.v02 --- sata_sat.v03 --- sata_sat.v04 --- scsi_aac.v00 --- scsi_adp.v00 --- scsi_aic.v00 --- scsi_bnx.v00 --- scsi_bnx.v01 --- scsi_fni.v00 --- scsi_hps.v00 --- scsi_ips.v00 --- scsi_lpf.v00 --- scsi_meg.v00 --- scsi_meg.v01 --- scsi_meg.v02 --- scsi_mpt.v00 --- scsi_mpt.v01 --- scsi_mpt.v02 --- scsi_qla.v00 --- scsi_qla.v01 --- uhci_usb.v00 --- tools.t00 --- xorg.v00 --- imgdb.tgz --- imgpayld.tgz
build=
updated=0

vboot.c32, however, expects to find boot.cfg in the tftp root directory. This is usually the parent to your pxelinux.cfg directory. That kernel file (tboot.b00) and those modules (b.b00, jumpstrt.gz, etc.) possibly don’t live in your tftp root, so you need to specify the path to all of those files relative to tftp root. In my environment, I have a subdirectory called esxi55u2, so boot.cfg looks like this:


bootstate=0
title=Load ESX Installer
kernel=/esxi55u2/tboot.b00
kernelopt=runweasel
modules=/esxi55u2/b.b00 --- /esxi55u2/jumpstrt.gz --- /esxi55u2/useropts.gz --- /esxi55u2/k.b00 --- /esxi55u2/chardevs.b00 --- /esxi55u2/a.b00 --- /esxi55u2/user.b00 --- /esxi55u2/sb.v00 --- /esxi55u2/s.v00 --- /esxi55u2/ata_pata.v00 --- /esxi55u2/ata_pata.v01 --- /esxi55u2/ata_pata.v02 --- /esxi55u2/ata_pata.v03 --- /esxi55u2/ata_pata.v04 --- /esxi55u2/ata_pata.v05 --- /esxi55u2/ata_pata.v06 --- /esxi55u2/ata_pata.v07 --- /esxi55u2/block_cc.v00 --- /esxi55u2/ehci_ehc.v00 --- /esxi55u2/elxnet.v00 --- /esxi55u2/weaselin.t00 --- /esxi55u2/esx_dvfi.v00 --- /esxi55u2/xlibs.v00 --- /esxi55u2/ima_qla4.v00 --- /esxi55u2/ipmi_ipm.v00 --- /esxi55u2/ipmi_ipm.v01 --- /esxi55u2/ipmi_ipm.v02 --- /esxi55u2/lpfc.v00 --- /esxi55u2/lsi_mr3.v00 --- /esxi55u2/lsi_msgp.v00 --- /esxi55u2/misc_cni.v00 --- /esxi55u2/misc_dri.v00 --- /esxi55u2/mtip32xx.v00 --- /esxi55u2/net_be2n.v00 --- /esxi55u2/net_bnx2.v00 --- /esxi55u2/net_bnx2.v01 --- /esxi55u2/net_cnic.v00 --- /esxi55u2/net_e100.v00 --- /esxi55u2/net_e100.v01 --- /esxi55u2/net_enic.v00 --- /esxi55u2/net_forc.v00 --- /esxi55u2/net_igb.v00 --- /esxi55u2/net_ixgb.v00 --- /esxi55u2/net_mlx4.v00 --- /esxi55u2/net_mlx4.v01 --- /esxi55u2/net_nx_n.v00 --- /esxi55u2/net_tg3.v00 --- /esxi55u2/net_vmxn.v00 --- /esxi55u2/ohci_usb.v00 --- /esxi55u2/qlnative.v00 --- /esxi55u2/rste.v00 --- /esxi55u2/sata_ahc.v00 --- /esxi55u2/sata_ata.v00 --- /esxi55u2/sata_sat.v00 --- /esxi55u2/sata_sat.v01 --- /esxi55u2/sata_sat.v02 --- /esxi55u2/sata_sat.v03 --- /esxi55u2/sata_sat.v04 --- /esxi55u2/scsi_aac.v00 --- /esxi55u2/scsi_adp.v00 --- /esxi55u2/scsi_aic.v00 --- /esxi55u2/scsi_bnx.v00 --- /esxi55u2/scsi_bnx.v01 --- /esxi55u2/scsi_fni.v00 --- /esxi55u2/scsi_hps.v00 --- /esxi55u2/scsi_ips.v00 --- /esxi55u2/scsi_lpf.v00 --- /esxi55u2/scsi_meg.v00 --- /esxi55u2/scsi_meg.v01 --- /esxi55u2/scsi_meg.v02 --- /esxi55u2/scsi_mpt.v00 --- /esxi55u2/scsi_mpt.v01 --- /esxi55u2/scsi_mpt.v02 --- /esxi55u2/scsi_qla.v00 --- /esxi55u2/scsi_qla.v01 --- /esxi55u2/uhci_usb.v00 --- /esxi55u2/tools.t00 --- /esxi55u2/xorg.v00 --- /esxi55u2/imgdb.tgz --- /esxi55u2/imgpayld.tgz
build=
updated=0


VMware vSphere ESXI 5.5 PXE Install

vboot.c32 options

VMware’s special version of mboot.c32 has command line options not available in SYSLINUX mboot.c32. These options are mentioned in the source code. These should clue you into the possibilities. My comments are in italic font.

  • -a Do not pass extended attributes in the Multiboot info memory map. This is necessary for kernels which do notsupport Multiboot memory map extensions, and which hardcode the memory map entry size.
  • -c {FILEPATH} Set the configuration file to FILEPATH. Default is boot.cfg. DO NOT USE NON-ALPHA CHARACTERS except the dot
  • -S <1...4> Set the default serial port (1=COM1, 2=COM2, 3=COM3, 4=COM4).
  • -s {BAUDRATE} Set the serial port speed to BAUDRATE (in bits per second). Default seems to be 115200
  • -t {TITLE} Set the bootloader banner title.
  • -R {CMDLINE} Set the command to be executed when is pressed. is only executed if the underlying firmware library does not allow to return from main().
  • -p 0...n Set the boot partition to boot from.
  • -e Exit on transient errors.
  • -V Enable Verbose mode.

I haven’t experimented extensively with these besides the -c option to specify the config file, and the serial output options (which is interesting in a manufacturing and test environment).

What can I do with boot.cfg?

-c is used to specify a config file other than boot.cfg. In my environment, I can select different configuration files to do an unattended install, for serial output, and even a diskless network boot. The note about non-alpha characters is important! Something as innocuous as specifying a file name like “boot-5-5-U2.cfg” breaks the install in weirdly mysterious ways. To avoid trouble, I’ve stuck with old fashioned MS-DOS 8.3 file names with all lower case letters. I dislike drama so I’ve avoided experimentation with file names beyond that.

For a diskless network boot, just remove the line kernelopt=runweasel.

For serial output, add this to your kernelopt line:


gdbPort=none logPort=none tty2Port=com1 com1_baud=9600

Finally, for automated install, add to your boot.cfg a location for a kickstart file to kernelopt, so it looks something like this.


kernelopt=runweasel ks=http://pxeserver/pxeroot/esxi55u2/ks.cfg

I haven’t tested it, but I imagine tftp is supported as well as http by the install program.

VMware Kickstart

The ks.cfg Kickstart file is based on the regular Redhat Kickstart. There are important differences between Redhat’s Kickstart and VMware, so don’t just copy a Redhat Kickstart file and hope it works. You should build your own ESXi ks.cfg from scratch. VMware documents there Kickstart options here. Here’s my ks.cfg for an unattended install, which accepts the VMware eula, sets a root password, overwrites every disk on the system, installs to the first disk, and specifies vmnic0 as the management interface. The %firstboot section enables ssh and shell logins so you don’t have to go through three levels of menus to configure that manually.


vmaccepteula
rootpw vmware123
clearpart --alldrives --overwritevmfs
install --firstdisk --overwritevmfs
network --device=vmnic0 --bootproto=dhcp
reboot

%firstboot --interpreter=busybox
vim-cmd hostsvc/enable_ssh
vim-cmd hostsvc/start_ssh
vim-cmd hostsvc/enable_esx_shell
vim-cmd hostsvc/start_esx_shell

%post --interpreter=busybox --ignorefailure=true

%pre --interpreter=busybox

I use a SYSLINUX version other than 3.86. Help!

Maybe you use a newer (or older) pxelinux.0 than 3.86 because UEFI, or compatibility, etc. Here’s the solution I use: grab syslinux-3.86 from VMware’s vSphere Server open source tarball linked above. Edit the file server/syslinux-3/syslinux-3.86/core/pxelinux.asm and search for this line:


default_str db 'default', 0

Replace 'default' with 'vefault'. This is the default PXE configuration file.

Type “make” in the syslinux-3.86 directory. You might have errors, depending on your development environment, but as long as you can create the files core/pxelinux.0 and com32/menu.c32, you should be golden.

Rename pxelinux.0 to vxelinux.0 and menu.c32 to venu.c32 and copy them to the tftp root on your PXE server. These live alongside the existing pxelinux.0 and menu.c32. You also create a new default file in pxelinux.cfg, except you name it “vefault”. The default binary (usually the very first line) in vefault should read:


DEFAULT venu.c32

Add your VMware install selections below that.

Finally, you add an entry to your normal (non-version 3.86) PXE menu that chains to your new, VMware-only vxelinux.0 that looks something like this (depending on the version of pxelinux you use)


label VMware
menu label VMware install
COM32 pxechn.c32 pxeserver::/vxelinux.0

Questions –> Please feel free to comment below.

8 Comments

  • Pingback: Загрузочный сервер — как загрузочная флешка, только сервер и по сети - itfm.pro

  • Pingback: Загрузочный сервер — как загрузочная флешка, только сервер и по сети | Валентина Драйчикова, Английский, Немецкий в Астане, Репетитор в Аст

  • David Kennedy
    June 24, 2015 - 8:05 am | Permalink

    This is an excellent guide. What I would reiterate the Open Source you want is for vCenter not vSphere when you download look for the syslinux-3 folder for the tar.gz containing the files to recompile. The portal has changed somewhat since this guide was published.

    I found myself needing to fully recompile the source because Windows Deployment Services version of TFTP does not append the correct paths for that version of SysLinux so I had to change the pxelinux.asm Line 2877:

    cfgprefix db ‘boot\x64\pxelinux.cfg\’

    I also had to adjust the registry keys to allow for front slashes to be accepted by their version of TFTP on WDS for Windows Server 2012.

    That being said you can also do the change to vefault WITHOUT recompiling if you are using TFTP on linux simply by using a HEX editor like xvi32 on windows, find the word default and change the d to a v.

  • August 3, 2015 - 2:49 am | Permalink

    Thats an excellent post and thanks a lot for posting this. It saved me a lot of time after trying different versions of syslinux. i am stuck on the unattended/scripted install. For some reason when i select the scripted install, it doesnt load the KS.cfg file or sometimes only accepts vmware eula and avoids the rest. I am using windows 7 with TFTPD32 and filezilla . Any help/advise appreciated.

    Thanks

  • August 4, 2015 - 2:45 pm | Permalink

    @venkata: I’m not at all familiar with Windows so can’t suggest much. You may have checked these already and I don’t know if they even apply, but things that come to mind:

    * file permissions: does your tftp server have read permission on your ks file?
    * filename munging: I don’t think this is a thing for Windows anymore, but maybe there’s something funny going on with lower case vs upper case on the file name?
    * Vmware’s mboot loader (which is what loads the kickstart file) has problems with non-alpha characters in the path / filename. I haven’t tested but it may fail on simple things like upper case characters or non-Unix style path (e.g. the reverse slash often used by Windows admins to specify NFS paths).

    Good luck and let me know what turns up if you solve this.

    Also, I figured out how to make network install in UEFI mode to work. Let me know if anyone is interested in this.

  • Pat
    August 19, 2015 - 12:43 pm | Permalink

    are you able to PXE install under UEFI using syslinux.efi + mboot.c32 ? I’d like to see that, Thanks.

  • August 19, 2015 - 12:52 pm | Permalink

    Yes, I have esxi 5.x and 6.0 network boot working on systems in UEFI boot mode.

    On your VMware ISO is a file called bootx64.efi. Instead of syslinux.efi, you need to chain to this vmware-provided bootx64.efi (i.e. load and execute this file). Command line parameters are exactly the same as the VMware-provided mboot.c32 program (they’re compiled from the same source).

  • Pat
    August 20, 2015 - 3:00 am | Permalink

    thanks Richard !

  • Leave a Reply