PEP 750: Tag Strings for Writing Domain-Specific Languages
discuss.python.org
discuss.python.org
The concern isn't having features that will make it easier to write DSLs, my problem is that people will misuse it in regular Python projects.
I know that one of the authors are Guido, but I'm not buying the motivation. Jinja2 and Django template are pretty much just using Python, it's not really much of an issue, and I don't believe that business logic should exist in your templates anyway. As for the SQL argument, it will still be possible for people to mess it up, even with Tag Strings, unless you completely remove all legacy code. The issue here isn't that the existing facilities aren't good enough, it's that many developers aren't aware of the concepts, like prepared statements. If developers aren't reading the docs to learn about prepared statements, why would they do so for some DSL developed using tag strings?
Obviously Guido is a better developer than me any day, so I might be completely wrong, but this doesn't feel right. I've seen tools developed to avoid just doing the proper training, and the result is always worse.
Because you've deprecated the "bare string" interface so they can't use that anymore, or it's hidden deep into the utility modules.
Someone else also pointed out that you could just do this with functions. It seems like a very fancy way of avoiding using (). I don't know, maybe show me how that would solve the issue of unsafe SQL and I'd be more easily convinced.
This already exists in the Javascript ecosystem:
sql`select * from users where id = ${id}`
{ query: 'select * from users where id = $1', values: [id] }
sql`select * from users ${'where id = 3'}`
{ query: 'select * from users $1', values: ['where id = 3'] }
Examples:
- slonik: https://github.com/gajus/slonik?tab=readme-ov-file#protectin...
- postgres.js: https://github.com/porsager/postgres?tab=readme-ov-file#quer...
There’s Python the scripting language, replacement for bash scripts, R and Lua.
Then there’s Python the serious software development language. Where people have style guides, code review, test coverage, and projects with more than one directory.
I understand why people in the second group are fearful of this feature and DSLs in general. I’m in the first group and I’m quite excited for it.
I'd be very interested in seeing where this goes, it certainly has it's uses, but there's also a ton of projects where it really doesn't belong.
Our Python users' Slack channel at work is already overcrowded with messages to the effect of, "halp what's this syntax how does this code work."
Python is very popular for machine learning, in days gone past ML researchers used the Torch library in Lua. TensorFlow and PyTorch made Python the dominant language in that field.
regex -- DSL for search/replace in text. Useful in moderation
jq -- DSL for search/replace in json. Useful on the command-line
xpath -- DSL for searching trees (hierarchy)
sql/xslt -- I would rather read Python most of the time instead but sometimes it is helpful to have an option of writing SQL directly (pep may help)
toml/json -- writing Python directly is preferable (in the context of .py file)
markdown -- DSL for writing markup (instead of html/word). I wouldn't say no to inline docstring rendering in Python code (pep may help). The same for (subset of) latex -- for math formulas `$e^{i \pi} -1 = 0$`
dot/plantuml (ebnf, sequence diagrams, etc) could be useful for literate programming-heavy style.
I see things differently. In those, nearly everything is a parallel universe where things aren't Python. Scope rules. Imports. Control flow. Calling things. Navigation. Refactoring. Linting. Therefore, Python tooling can't really help much. Tools have to write special rules to help on each template language and framework.
Instead, imagine if Black, Ruff, mypy, pyupgrade, and IDEs could treat the template part as software.
I realize I'm describing "people who want Python tooling", which might be a minority. I also realize there's pushback about additions to Python. That said, I think TSX and friends have shown -- there's lots we can do to improve Python web development.
That f-strings were "static" was by far my biggest criticism of it, given how useful I find JS's template strings.
And this proposal seems like a straight up better version of template strings:
- the static strings and interpolations are not split and don't have to be awkwardly re-interpsersed which I've never found 100% trouble and 0% utility
- the lazy evaluation means it can be used for things like logging (which really want lazy evaluation), or meta-programmation (because you can introspect the callables, and you get the expression text)
Could you elaborate? I would find it rather surprising if my log messages don't contain the data at the very moment I invoke the logger.
In most APIs this requires an explicit conditional check and the average developer will not think of it. This allows said check to be performed internally.
It's such a subtle difference that I only notice it when my IDE underlines it :/
html() is not going to be equivalent.
Companion repo with: JupyterLite playground, Docker images, other material. https://github.com/pauleveritt/tagstr-site
It required a janky workaround for the absence of this PEP: https://pypi.org/project/tagged/
I used these to investigate ideas about component-driven development: https://viewdom.readthedocs.io/en/latest/
- No tooling in Python will do anything with the string, so DX suffers. - Evaluating the variable requires frame evaluation, which is...problematic.
You could do `html(f'<div>Hello {name}</div>')` and get f-string coding assistance. But you'd also get immediate evaluation. There's nothing the `html` function can do with the `{name}` part.
At the risk of riling some people up, this smells like Perl. Some poor Python user comes across
greet"Hello {user}"
But I guess Python and I parted ways on this matter quite a while ago. I like to joke about Katamari Dama-C++ but Python is starting to give it a run for its money. C++ is still in the lead, but Python is arguably sustainably moving more quickly on the "add more features" front.
Also it’s harder to do nothing than something.
That being said, I think this is partly abstract. I’ve just ignored a lot of new Python features without issue. And while I worried that they’d force me to learn new things to understand others’ code, that’s not really materialized.
That identifier has to come from somewhere. In order for this to work:
greet"Hello {user}"
from some_tag_strings_library import greet
But in Python I could also imagine YET ANOTHER constant prefix, like t"", that returns a "template" object of some sort, and then you could do html(t"") or whatever. That is, it would be just like f"" but return an object and not a string so you could get at the underlying values. Like in JavaScript the ability to see the original backslash escaping would be nice, and as an improvement over JavaScript the ability to see the expressions as text would also be nice.
But the deferred evaluation seems iffy to me. Like I can see all the cool things one might do with it, but it also means you can't understand the evaluation without understanding the tag implementation.
Also I don't think deferred evaluation is enough to make this an opportunity for a "full" DSL. Something like a for loop requires introducing new variables local to the template/language, and that's really beyond what this should be, or what deferred evaluation would allow.
I don't particularly mind the prefix thing. It came up in the PEP discussion, as did choice of backticks to indicate this is different. But JS template literals -> tagged template literals shows, you can get from A to B without a fundamental change.
I'm very interested though in the deferred part. I agree that there is complexity. I weigh that, though, against the complexity of existing Python HTML templating, where finding out what just happened is...harder.
I think we can get a TSX-level of DX out of this. And maybe a Lit level of composition. Agree that it is non-zero complexity.
I think JSX is an example of the somewhat crude but practical use of simple execution patterns. For instance if you have a loop you do:
return <ol>
{items.map((item, i) => <li key={i}>{item}</li>)}
</ol>;
Or in a SQL builder with JavaScript tagged templates, I do:
exec(sql`
SELECT * FROM a_table
WHERE category = ${category}
${subcategory ? sql`AND subcategory=${subcategory}` : sql``}
`)
If there's deferred execution, it's done with ?: and .map() – though these very long expressions don't work nearly as well in Python. (List comprehension is in some ways better than .map()/.filter(), but not for very large expressions like in a JSX template.)
It just doesn't seem worth it to define a whole new thing just to abstract over a format() function call. The laziness might be interesting, but I feel like "lazy strings" might be all that's needed here. Laziness and validation (or custom string formatting logic) are separate concerns and should be separated.
That could also be leveraged at f-strings themselves.
> Laziness and validation (or custom string formatting logic) are separate concerns and should be separated.
In which case the one to move out is the laziness not the customised interpolation. Because the latter is the one that's necessary for safer dynamic SQL or HTML or whatever.
If a syntax such as f"{variable}" is already a feature - and turned out to be a popular one - why shouldn't we be able to add our own custom "f"s? Because that is what this is about. It might make generating output even simpler.
I applaud the idea and am pleased to see that Python keeps innovating!
greet("Hello {name}")
What was wrong with the standard way to write function application?
Python is sufficiently dynamic that an implementation of greet(...) can look up one level to resolve {name}, right? That's why Python will forever run like a dog. Might as take advantage of it to build such capabilities in user space.
This crap is going to end up inside f-strings inside tag-strings inside f-strings inside... We have a language. Don't extend it to express what it's perfectly capable of expressing already.
> Tag strings extract more than just a callable from the Interpolation. They also provide Python string formatting info, as well as the original text.
The feature is akin to moving print from a keyword to a function. That change also made a huge difference in that it unified the output stream and avoided having undefined objects like a "print" keyword.
Here, you can think of the feature as moving an "f" string from a hardcoded, predetermined definition to a generalizable and programmable behavior.
If "f" strings have become so popular so quickly it means they addressed a pressing need. It is logical to assume that a programmable version of an "f" string would be even more useful.
From their POV, inside the quotes, what you know about f-strings, you know here as well.
> ...other stuff that happens before evaluation...
A greet(string) function could parse the string and resolve the names itself:
parsed = parser(string)
resolved = resolver(parsed)
return formatter(resolved)
If you hate boilerplate, make the first two steps into a decorator.
A PEP introducing a grand unified theory of magic (tag strings) isn't inherently better than the status quo of some (f-string) magic. Less magic is better.
If the string is not an f-string, you get no help from Python tooling.
In both cases, you have to use frame hacks to get back to the scope, which has negative consequences.
So? It's been evaluated successfully. What more is there to do?
> If the string is not an f-string, you get no help from Python tooling.
Expose that tooling via the standard library. It's just pure functions.
> In both cases, you have to use frame hacks to get back to the scope, which has negative consequences.
What consequences? Isn't CPython forced to do all the nasty stuff anyhow when it's a language feature?
Instead, why not add a single new string prefix, like "l" for "lazy"? So, f"hello {name}" would immediately format it while l"hello {name}" would produce an object which contains a template and the captured variables. Then their example would be called like: greet(l"hello {name}").
@Language("application/sql")
String query = "SELECT 1";
@Language("application/graphql+json")
String query = """
query HeroNameAndFriends {
hero {
name
friends {
name
}
}
}
""";Also the scoping rules of this are a special case which doesn't appear in regular Python code so far: "The use of annotation scope means it’s not possible to fully desugar interpolations into Python code. Instead it’s as if one is writing interpolation_lambda: tag, not lambda: tag, where a hypothetical interpolation_lambda keyword variant uses annotation scope instead of the standard function scope." -- i.e. it's "as if you wrapped all interpolation expressions in a lambda: <expr>, except it uses different scoping rules".
Compared to what?
At the end of the day you're still doing string formatting, if you want parsing, then you'd feed the item into a parser, which this doesn't preclude.
The interface sounds a lot better than JS's anyway, as that completely separates the literal strings and the interpolations so you have to re-intersperse them which is muggy.
> interpolations become four-tuple-like
They become an Interpolation object, which can be unpacked if you find that more convenient, but you can access the members if you prefer:
- 0 is getvalue is the callable to retrieve the evaluated expression
- 1 is expr is the raw text form of the expression
- 2 is conv is the !conversion tag (s, r, or a)
- 3 is format_spec
1. Parse to an AST, generally using an off-the-shelf parser. In practice, it's possible to rewrite interpolations with a placeholder suitable for a given language, eg x$Nx for HTML. Of course if that doesn't actually work, you might have to write/modify an existing parser. Hopefully we can cleverly avoid this extra work.
2. Walk/compile the AST, filling interpolations, but taking in account the context. This can for example take the form of building appropriate query strings that avoid Bobby Tables SQL injection, whether by mapping to SQL placeholders or with appropriate quoting (such as for a column or table name).
3. Memoize these steps, much as we see with builtin DSLs in Python, like the re module; see https://github.com/python/cpython/blob/3.12/Lib/re/__init__.... We do plan to make this easier/faster by supporting getting the original source of the template string (Template.source), vs the *args approach we show in the PEP at the start of this discussion (this will become Template.args instead; Template here is the proposed protocol of the object passed in).
Related is my post here: https://discuss.python.org/t/pep-750-tag-strings-for-writing...
I hope they eventually grant this power to regular functions because otherwise I know folks will end up myfunc"{arg1},{arg2}" to get that feature.
def greet(*args: Decoded | Interpolation) -> str:
result = []
for arg in args:
match arg:
case Decoded() as decoded:
result.append(decoded)
case Interpolation() as interpolation:
value = interpolation.getvalue()
result.append(value.upper())
return f"{''.join(result)}!"
def greet(*args):
def convert(arg):
return arg.getvalue().upper() if hasattr(arg, getvalue) else arg
return ''.join(convert(arg) for arg in args) + '!'The pattern matching stuff is neat but seems pretty half-baked. Only available to imperative code and restricted in what can be matched. I wish they'd finish fleshing it out because right now it feels tacky.
Actually my number one wish forever has been that functions would become actual first-class objects, able to be defined and manipulated during runtime at the syntax level. Would have been great for so much of my code, and the original decorator module. Things are better with the new inspect module but it's still ridiculously clunky.
1. https://protostrings.readthedocs.io/en/latest/index.xhtml
iirc I wanted to encode state in a recursive-descent parser without additional complexity to the parser.
Similar in purpose not design; protostrings provides lazy and context-sensitive strings from the bottom up rather than this top-down template-style proposal, which I feel addresses most of the concerns here.
- tags get a strings array that's referentially stable across invocations. This can function as a cache key to cache strings parsing work. - tags can return any kind of value, not just a string. Often you need to give structured data to another cooperating API.
Deferred evaluation of expressions is very cool, and would be really useful for reactive use-cases, assuming they can be evaluated multiple times.
Tags can return any kind of value, so there's that.
Deferred evaluations can be evaluated multiple times and is in fact one of the biggest foot-guns in this API (in my opinion).
The JS API where the strings are separate might seem awkward at first, but it ends up being a really elegant design.
Deferred evaluation is really powerful and I wish JS had it. It's one of the reasons why Solid has a custom JSX compiler and doesn't use tagged literals... to get the-evaluation you need to user to pass in closures, which is cumbersome.
Template.source can act as a memoization key.
Most of the critiques I am reading (with which I fully agree) is that this is more complexity in the language without sufficient payoff. There are now how many stupid or fancy ways to construct a Python string with a variable?
Moreover, Python has resume-driven development and people with a financial or reputational interest will get their new toys in no matter what.
You would just contribute to the appearance of democracy.
Another kitchen sink to add to Python's world-class kitchen sink collection.
greet("hello {world}")
If so: why not just do that? I would expect the performance to be kinda terrible compared to a language-intrinsic, but this hardly seems like a thing worth truly optimizing. And if it's too costly at runtime, someone can implement a parse-time AST rewriter, like many Python things already do. Heck, that example's `assert` is probably using pytest (because everyone uses pytest) and it's doing exactly this already, and it isn't a language feature, it's just a normal library using normal Python features.
It's been nearly 23 years, but Python 2.2 fixed this issue (https://docs.python.org/3/whatsnew/2.2.html#pep-227-nested-s...), and it's also why JavaScript added let (and const). f-string support also uses lexical scope, and it's an important part of its success.
From the article: """
~CURL[curl https://catfact.ninja/fact]
|> Req.request!()
So I remain doubtful, as long as no way is shown to me, how the user of the language can define this syntactic abstraction themselves, which is unlikely to happen or exist in Python.