Skip to content

Instantly share code, notes, and snippets.

@FlorianHeigl

FlorianHeigl/boot.uscr

Last active November 8, 2025 14:05
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save FlorianHeigl/f818d6a50667d7bc134418e92a6ade98 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save FlorianHeigl/f818d6a50667d7bc134418e92a6ade98 to your computer and use it in GitHub Desktop.
Download ZIP
Alpine Linux ConnectX-3, building mstflint 4.22
Raw
boot.uscr
setenv xen_addr_r 0x42e00000
load mmc 0 ${xen_addr_r} /boot/xen
load mmc 0 ${fdt_addr_r} /boot/dtbs/${fdtfile}
load mmc 0 ${kernel_addr_r} /boot/vmlinuz-sunxi
fdt addr ${fdt_addr_r}
fdt resize
fdt set /chosen \#address-cells <1>
fdt set /chosen \#size-cells <1>
fdt mknod /chosen module@0
fdt set /chosen/module@0 compatible "xen,linux-zimage" "xen,multiboot-module"
fdt set /chosen/module@0 reg <${kernel_addr_r} 0x${filesize} >
fdt set /chosen/module@0 bootargs "modules=loop,squashfs,sd-mod,usb-storage clk_ignore_unused rootflags=size=1000M modloop=/boot/modloop-sunxi modloop_verify=false console=hvc0"
fdt set /chosen xen,xen-bootargs "dom0_mem=1400M loglvl=all guest_loglvl=all no-bootscrub dom0_max_vcpus=1 global-lock=0 vwfi=native"
load mmc 0 ${ramdisk_addr_r} /boot/initramfs-sunxi
bootz ${xen_addr_r} ${ramdisk_addr_r}:${filesize} ${fdt_addr_r}
Raw
build-notes.md

took notes from proxmox post

relevant bits for building older mst that works with connectx-3

wget http://www.mellanox.com/downloads/firmware/fw-ConnectX3-rel-2_42_5000-MCX312A-XCB_A2-A6-FlexBoot-3.4.752.bin.zip https://www.mellanox.com/downloads/firmware/mlxup/4.22.1/SFX/linux_x64/mlxup https://www.mellanox.com/downloads/MFT/mft-4.22.1-11-x86_64-deb.tgz https://github.com/Mellanox/mstflint/releases/download/v4.22.0-1/mstflint-4.22.0-1.tar.gz tar -xvzf mstflint-4.22.0-1.tar.gz

packages installed

5 apk add sudo 13 apk add apk-tools-static 20 apk add bash ipmitool openipmi 24 apk add nvi 25 apk add vi 26 apk add vim-minimal 27 apk add vim 28 apk add lldpd 35 apk add mstflint 36 apk add mlx 38 apk add rmda-core-libs 39 apk add rmda-core-libs-57 40 apk add rdma-core-libs 43 apk add rdma-core 44 apk add infinib 48 apk add lspci 49 apk add pciutils 56 apk add ethtool 76 apk add alpine-sdk bison flex 82 apk add linux-headers 85 apk add gcompat 96 apk add zlib-dev 97 apk add openssl-dev 105 apk add pciutils-dev pciutils-libs 109 apk add libpciaccess-dev libpciaccess 112 apk add libpciaccess 113 apk add syslinux 117 apk add syslinux-dev 123 apk add musl-dev 131 apk add ccache 140 apk add findutils 207 apk add rdma-core-dev

build errors, fixes/patches

fixs variable declarations in all of these like this:

sed -i -e 's/u_int32_t/uint32_t/g' -e 's/u_int16_t/uint16_t/g' -e 's/u_int8_t/uint8_t/g' -e 's/u_int64_t/uint64_t/g'

  • mtcr_ul_com.c
  • packets_layout.h
  • common/compatibility.h
  • kernel/mst.h
  • include/mtcr_ul/*.h
  • mtcr_ul/*h
  • mtcr_ul/*c

comment duplicate definitions in a fucking file i can't put on my clipboard because macos in 2025 can't even do copy paste anymore you goddam stupid losers if not for security issues I would immediately go bad to OSX 10.4, those maniacs have improved nothing, do you have any idea how much faster it was back then? memory compression my ass.

so, anyway, in mtcr_ul_com.c , lines: 87, change sys/pci.h to linux/pci.h 393-397, comment out PCI_DEVFN and PCI_FUNC declarations, they are in libpcistuffdev

./configure && make && make install

this installs to /usr/local so no conflict with the alpine package the proxmox person had to do ./configure --disable-inband but with rdma-core-dev installed you can leave it included. it's needed for managing IB cards / switches over inband channels (inband here is not a short for infiniband). that function matters a lot for unbricking.

after it's installed, go to /usr/local/bin find your card using lspci

x10:/usr/local/bin# lspci | grep -i mellanox
04:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

if you're lucky it'll be in slot 04, then you can copy paste their examples!

here we go:

x10:/usr/local/bin# ./mstconfig -d 04:00.0 q

Device #1:
----------

Device type:    ConnectX3Pro    
Device:         04:00.0         

Configurations:                                      Next Boot
         SRIOV_EN                                    True(1)         
         NUM_OF_VFS                                  32              
         LOG_BAR_SIZE                                3               
         BOOT_OPTION_ROM_EN_P1                       True(1)         
         BOOT_VLAN_EN_P1                             False(0)        
         BOOT_RETRY_CNT_P1                           0               
         LEGACY_BOOT_PROTOCOL_P1                     PXE(1)          
         BOOT_VLAN_P1                                1               
         BOOT_OPTION_ROM_EN_P2                       True(1)         
         BOOT_VLAN_EN_P2                             False(0)        
         BOOT_RETRY_CNT_P2                           0               
         LEGACY_BOOT_PROTOCOL_P2                     PXE(1)          
         BOOT_VLAN_P2                                1               
         IP_VER_P1                                   IPv4(0)         
         IP_VER_P2                                   IPv4(0)

turns out SR-IOV was fully enabled so I didn't need to do any of this and the only thing gained are these notes and patches. what a surprise. note how this is not a VPI card, so there's no IB/EN/VPI mode setting. OK for me I don't do much infiniband anymore since almost no software defined storage devs support it. They rather seem to hate it though it would be much faster for them, especially for 32/64K IO sizes it would be better... ROCE luckily also works so whatever. Oh, what I use also doesn't support ROCE and KVM also just "has patches" and if you to try them it's

  • nightmarish
  • unoptimized and hardly faster

so all you gain is less cpu overhead for not saturating the link, whereas you should gain no cpu utilization and almost instant migrations. well. lol.

back to making sr-iov work on this test system and then find out why it doesn't work on luxury car-priced bull servers.

@FlorianHeigl
Copy link
Author

FlorianHeigl commented Nov 4, 2025

links

https://forum.proxmox.com/threads/enabling-sr-iov-for-intel-nic-x550-t2-on-proxmox-6.56677/

https://forum.proxmox.com/threads/how-to-configure-mellanox-connectx-3-cards-for-sriov-and-vfs.121927/#post-792768

https://forums.servethehome.com/index.php?threads/help-with-connectx-3-sr-iov-with-linux-host-and-windows-guest-via-kvm.28956/page-3#post-454184

(warning, windows guest driver problems, see patches in thread, author has not upstreamed due to lack of QA/cleanup

@FlorianHeigl
Copy link
Author

FlorianHeigl commented Nov 4, 2025

This outlines how to actually make it usable afterwards:
https://gist.github.com/FlorianHeigl/a7b307e666fd471981f5d2e823f12cf4

@FlorianHeigl
Copy link
Author

FlorianHeigl commented Nov 4, 2025

you can check your FW version and PSID with mstflint (once you got the old-enough-version)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment