It isn't a dumb and simple program. It does a lot of complicated stuff under the hood, and it is good to be aware of that.
The actual problem here is that such a dumb simple task as parsing an ELF file isn't something you'll do without vulnerabilities, if you're not paying attention to vulnerabilities. Run `man elf` on your favorite Linux system, and you'll see docs of <elf.h>, which you can use to read ELF files. Write something to just dump the text section -- it shouldn't take you more than like 30 lines of C. Then tell me how many integer overflows you managed to squeeze in those 30 lines. :)