Any suggestions for engineering observability?
In terms of implementation, if it isn't on a dashboard or doesn't fire off an alert, it doesn't exist. So put your telemetry and logs on a dashboard, and set up alerts based on your SLA's/SLO's.
Also, make it easy for developers to understand, create and edit dashboards/alerts. If it's hard to understand or hard to do, it won't get done.
That may well change during development, but new work doesn't get approved without at least a first stab at how we're going to monitor it.