+ Use structured logging (ex: https://www.structlog.org/en/stable/why.html)
Edit: Good lord people, set the collector to never sample and use one of the many file exporters and you get completely bog standard logs but with a huge amount of work already done for you and enough context to follow complicated multi-threaded/multi-process programs without making your head spin.
Tracing is usually sampled, or much more time limited, and aggregation tooling isn't nearly as good compared to logs with most providers. Much better for deep dives into a narrow context. For a single request drilldown or aggregated performance/error metrics, tracing every time.
Structured logging tends to give you rich search against broader contexts (e.g. long spans of time, many requests in aggregate, random non performance/error attributes you may care about) without the same concerns of data cardinality.
I felt nowhere near the same power with tracing tooling for random broad context as I've had with e.g. Splunk for logs. That's not to say you can't somehow have the best of both worlds, but I haven't seen it in the ecosystem just yet.
There are a few concepts to understand, but instrumenting it right from the start is valuable as it grows into a bigger system. Being a cross-cutting concern, the more code that standardizes on this the more useful it will be.
In the simplest case you could just output a sequence of values (for example as a csv, or one json array per line), instead of an interpolated string. The size and CPU overhead of this approach is minimal, but makes it much easier and more robust to parse the log afterwards.
Such basic structured logging is quite similar to what the article proposes in "Separate parameters and messages", but more consistent and unambitious if one of the arguments contains your separator character.
You almost certainly also have it in memory more than twice since you're copying a buffer out to IO.
If you care about record size use protobuf, compression, backend parsers, etc.
Of course some developers may just go ahead and create a JSON/CSV string from scratch, which is about as safe as string interpolation instead of parameter bindings for SQL statements /s. But provided they're using a proper JSON/CSV/proto object builder library that uses String references, no data is going to get copied until a logging component needs to write it to a file or a network buffer. Therefore the only overhead are the pointers that form the structure.
If you want to preserve the "one line = one record" property typically expected from logs, standard csv will not be suitable (since it doesn't escape linebreaks), but json without unnecessary linebreaks will handle that (known as json-lines).
"$TIMESTAMP $LEVEL: Hey look index $INDEX works!"
you have
$TIMESTAMP $LEVEL $FORMATTER $MSG $INDEX
I'm quite intentionally drawing a parallel to `echo` vs `printf` here, where printf has an interface that communicates intended format while echo does not.
The only overhead of structured logs done right is that you need to be explicit about your intended formatting on some level, but now with the bonus of reliable structure. Whether you use glog or spdlog or whatever is somewhat inconsequential. It's about whether you have an essentially unknown format for your data, having thrown away the context of whatever you wish to convey, or whether you embed the formatting with the string itself so that it can later be parsed and understood by more than just a human.
If you're concerned about the extra volume of including something like "[%H:%M:%S %z] [%n] [%^---%L---%$] [thread %t] %v" on every log entry, then you use eg (in GLOG equivalent for your language) LOG_IF, DLOG, CHECK, or whatever additional macros you need for performance.
If I'm wrong or just misunderstanding, please do correct me.
With structured logging the outputs may be trivial in which case the compiler can do a good job. But if you are inserting a std::string into a log line and your log output format is JSON, there's nothing the compiler can do about the fact that the string must be scanned for metacharacters and possibly escaped, which will be strictly more costly than just copying the string into the output.
Anyway.
Isn’t the choice of eg JSON-string vs any other string format somewhat beside the point? Wouldn’t you either 1) need to scan for metacharacters or 2) NOT need to scan for metacharacters, regardless of whether your log is structured or unstructured, at time of output?
Ignoring for a moment the additional cost per character of outputting additional data, but ignoring nothing else in the scenario, wouldn’t something like, for example:
“My index is {index}␟index=5”
cost the same to output as:
“My index is 5”?
It seems to me that the cost of interpreting the formatting doesn’t need to be paid until you wish to read the log message, which presumably you don’t strictly need to do at all until you actually care about the content of the message, or at the very least can defer the cost until a less critical moment, or do some “progressive enhancement” dependent on some set of pre-requisite conditions.
That may be the case, however is what you have now is "logging", I would 100% recommend the incremental upgrade to "structured logging".
I would not put an email address in logs, but definitely some kind of account identifier so that we can investigate bug reports from specific users.