I just found a nice reference that I remember using when working with Ubuntu ISOs before at https://help.ubuntu.com/community/LiveCDCustomizationFromScr...
It may be outdated and you may have already seen it, but the key point is the section "Install Packages Needed For Live System":
> apt-get install --yes ubuntu-standard casper lupin-casper
> apt-get install --yes discover laptop-detect os-prober
> apt-get install --yes linux-generic
Since they're building the ISO from scratch using debootstrap and a chroot, some scripts in the casper package will probably set up paths, grub, etc, all relative to the current root (the chroot they're working in).
This would also probably explain why using a vmlinuz.efi and initrd.lz from a default 16.04 iso kinda worked for you -- because some paths and versions match up, etc.
Long story short: you should try following that guide to some extent when packing up the custom.iso (unpack, install casper package(s) in a chroot, repack), then find a way to use grub on your ESP to "chainload" [1] the iso from the second partition.
HTH, I'll check back later and see if you had any luck :)
[1] https://wiki.gentoo.org/wiki/GRUB2/Chainloading#ISO_images