Just like you design software with the end user in mind, you should design security systems with the attacker in mind. Just like you prioritize features for by your expected user personas, you should prioritize security measures by your expected attacker personas. And just like you always want to cover the "basics" of UX, you always want to cover the "basics" of security.
First you should take care of the "low hanging fruit," meaning you should implement industry best practices at all levels of the stack. Hopefully this is sufficient to mostly mitigate any threats of getting caught in widely targeted attacks, like mass scans for Wordpress vulnerabilities.
Unfortunately, implementing security best practices is only the beginning of defending against motivated attackers focused specifically on you as a target.
Once you've taken care of the easy wins, you have to balance the tradeoffs of engineering effort and complexity required to make certain mitigations. In practice, this means that what your security system will look like largely depends on the question: What is your threat model?
If you are an IoT vendor, your threat model is becoming a node in a botnet. That means you need to defend against a focused attacker reverse engineering your product to find a vulnerability that can be used to root the device remotely. Unfortunately this is basically impossible to defend against, but you can make their job much more difficult with binary obfuscation, symbol stripping, aslr, etc. Your best hope at that point is the attacker gets frustrated and decides to target some other product.
If you're a search engine (cough cough) your threat model may be people scraping your search results (as hypocritical as that is...) In that case your first priority might be catching bots and serving up captchas.
Point is, your "security engineering" largely depends on your domain and threat model, just like "traditional" software engineering depends on domain and user base.
(That said, IANASecurityProfessional ;) )
If you're willing to incorporate it in the design from the outset, use specific components, and organize the project with that as a focus, you can make quality things.
Most things end up as an echo-y, impossible to hear cacophony as soon as there's many actors involved, because they just don't care and don't plan for it -- it would be a "waste of money", because it still performs economically without that.
You even have parallels like adding things to existing projects to try and fix them, and that not being nearly as effective than had you designed it better in the first place.
Systems are heterogeneous, and so are the exploits. A great example of this would be getting into a smart TV through broadcast signals (Weeping Angel).
A good place to start would be at the system level with a DREAD, DREAD-D, STRIDE or other model analysis. This is often a creative process, and experience counts.