A secure element chip does not need a completely libre/open design to support software freedom, but it does need to allow end-owners to run whatever software they would like - not subject to a third party's approval. To that end:
1. The chip needs to be able to be completely erased, having all keys zeroed out, at which time new keys can be loaded and/or internally generated. Signature verification keys are zeroed out to establish a new root of trust, and program/memory keys are zeroed out to maintain the security of information previously entrusted to the chip.
2. Resetting the chip should take a configurable amount of time (days to a month), either based on internally counting the clock or external proof of work, to protect against evil maid attacks by those with temporary possession of the hardware.
3. If there is any remote attestation functionality, then the reset procedure must output the attestation signing keys, including the private parts. This way the manufacturer (or any downstream integrators) cannot record the public identity of the shipped attestation keys to create a trust relationship that would prevent the ultimate owner from running software of their choosing.
For software freedom, these aspects are much more important than the design and licensing of the computation core(s). Let them ship ARM. Let them ship only ARM. Let them ship a proprietary ISA that needs to be painstakingly reverse engineered over a decade. Spend your social capital making sure the ultimate end-owners of the chip remain in control of their own devices! This is what is critical for open source.