This is an important topic to me as I've left several companies as an engineer over the years, some of which are rather well known.
Here are some suggestions simply as food for thought. Hopefully something applies in your case.
- Engineers should ideally be part of the process of building the product. If not already, invite engineers to the UI/Product design meetings and make sure they have active input - or at least one representative. The fact that engineers are in another building doesn't bother me so much when I think about the remote companies I've worked for. That being said, with major communication challenges it might require some initial face to face dialogue.
- Deadlines need to be realistic
Sometimes "simple" things are actually more complicated than one would think. I've seen scrum approaches that just don't make a lot of sense with unreasonable "story points" with strict adherence. Some flexibility needs to be in place to deal with unexpected blockers/issues. In one well known company a friend of mine indicated that they are super strict about story points and he hardly codes at all now since every engineer allocates only a low number of story points so that they can ensure they complete them before the end of the sprint.
- Better understand complexity of the product
It sounds like the product is "simple" but the reason for the large amount of code might be that there is no clear definition of the minimal viable product. The added complexity is possibly there because the UI/product team is piping in more requirements somehow. Try to isolate where the complexity is coming from.
- Promote a test driven culture
Regarding the "astounding" number of software defects, it doesn't seem like the engineering culture promotes unit testing. Consider adding TDD/BDD. Or if that already exists, track down what might be happening here. Maybe the software issues are mostly at the integration level?
- Promote a refactoring driven culture
I once worked at a company where management allocated time only once a year for a few weeks to deal with refactoring due to tech debt over the year. Needless to say there was way too much tech debt to pay off and those weeks were never enough! If the engineering leads say they need to address tech debt, let them have a strong voice in that at least. If time to market is more important so be it but be aware that tech debt can also mean the team is slower in the long run. Changeability of the code will slow everyone down over time significantly if it isn't addressed. For a large codebase we had, changing code was a nightmare because of this.
- Promote more than technical skills
Like most engineers I like to work with highly motivated, open-minded, approachable and friendly people. I've seen incredibly smart engineers stifle and essentially belittle other intermediate/junior engineers over their work or ideas. It just takes 1-3 of these smart engineers to cause bickering/analysis/paralysis in a project. I've joined projects in the past as a senior engineer where I've seen other intermediate/junior engineers quickly gravitate to me for example and feel comfortable asking questions of all sorts - I generally give a vibe of being easy going. My most memorable times at work revolve around tech questions/discussions with other engineers and I don't believe any of that was wasted time for me or the companies I've worked for. If I were to promote/hire a tech lead/architect I would try to ensure soft skill were in place rather than simply technical ability.
Usually management was not very aware the above conditions and and things can become 'normalized' enough such that nothing is ever done about improving things even if you spoke up. I've heard management say things like "we don't all have to be friends" and to just put up with the status quo but I think there is room for both.
- Give engineers a sense of ownership
I used to work at a place where the software architects essentially 'farmed' out JIRA tickets to engineers in such detail that there was no creativity whatsoever that the intermediate/junior engineers needed to have. Giving engineers the room to develop design skills and make mistakes is important.
- Allow and appreciate mistakes
Systems can get complex and fail for a wide variety of reasons. Personally if I were the engineering manager and an engineer brought down the system somehow I would raise that as a cause of celebration without the connotation of death ("post-mortem"). Celebration since this is one step further in figuring out how to make the system more robust. A lot of time cultures promote an environment of fear where people are afraid of changing the code because things break. Continuously improve things so that it is harder and harder to break.
- Create a culture of learning
More often than not, companies don't offer a great balance of "doing the work" and learning. In the past I've had to introduce the idea of tech talks, hackathons myself. Even purchasing books/videos at some companies was seen as an added cost that they didn't want to take on. Regarding tech talks, some companies allow engineers to create presentations during work hours and present them while in other companies I've had to put a lot of hours outside of work to prepare for presentations. Engineers will definitely appreciate the former if possible.
- Walk a mile in another person's shoes
In one company, engineers could sit and listen together with a support specialist once every few months to get a sense of what types of questions customers had about the product. Great eye opener.
Another idea is to allow shadowing of another person throughout the day or even do their job for a day if possible. An engineering manager could do a mini bug fix for example to get a sense of what is involved. I once had my manager's manager's manager put in a code fix in C++ - our team was super impressed not just for the tech skills but more for the fact that he didn't mind getting his hands dirty.
- Management should put their foot down when required
I recently worked at a company where there was a lot of swearing, sexual/racial jokes, etc during work hours as well as company events. The CEO was complacent in such things by saying "Oh, that's Joe, he's always putting his foot in his mouth".
I knew right there and then I would eventually leave. Sometimes management needs to take a stand somewhere.
Especially in the example you gave where there was some racism/sexual harassment if I were the CEO and learned about that I would drop everything I had and iron it out. I'm glad HR ironed that out in your company as you mentioned but I hope also the CEO came down with a message for everyone as well