Hacker News Reader

Comment by josevalim | Hacker News Reader

If you don’t mind, I have some questions as I would like to learn more. Are you aware of a large models.py for reference and learning purposes? Is it expected to define the schema of all my models in a single place but none of the logic?

Also, don’t you run into scenarios in Django where the automatic migration is not enough and you need to provide custom commands? In such cases, how do you provide them?

And can you have scenarios where you have two models pointing to the same table, but perhaps to a subset of fields (this is specially useful in read-only cases)? How is that handled?

Thank you :)

View on HN

Hi José, thanks for joining the discussion!

The sibling to this post (by “traverseda”) is more informed and informative than I could hope to write. I will add that Django has Manager classes that have default behavior included for interacting with the query API, and they can be customized to your needs. You can even have multiple managers for a given model.

https://docs.djangoproject.com/en/4.1/topics/db/managers/

It’s been a few years since I used Django, but I remember its model layer fondly, and still follow the framework’s release processes.

One of the areas where Django lags is being fully async (Python imposed a lot of limitations on async operations for most of its life). The upcoming 4.2 release of Django is laying the groundwork for async DB operations.

>Are you aware of a large models.py for reference and learning purposes? Is it expected to define the schema of all my models in a single place but none of the logic?

Django models are normal python classes. Depends on exactly the logic you're dealing with, but generally you can make logic be a method on that class. Try to avoid logic that spans multiple tables in general, and if you have logic that does span multiple tables you probably want it to be a function and not a model method.

There are also signals that get sent on different things like model delete/create/etc, but they're to be used even more sparingly.

Logic for querying data should probably go wherever you're going to use it and you should just pass the model objects directly to your views. To start don't worry about optimizing queries or the n+1 problem, but as you get more experience you can use `prefetch_related` in order to avoid the n+1 problem.

>Also, don’t you run into scenarios in Django where the automatic migration is not enough and you need to provide custom commands? In such cases, how do you provide them?

You shouldn't generally need to. That said django's migration system is very powerful and you can hand-write a migration if you need to.

>And can you have scenarios where you have two models pointing to the same table, but perhaps to a subset of fields (this is specially useful in read-only cases)? How is that handled?

I mean you can using proxy models but that's not really a thing with django. Models are for developers and generally developers have all the permissions any way, so the solution to making a model read only is to just not write to it. If you want to present a read-only model to an end user you can reference it in a view of make a read-only serializer with django-rest-framework. It's python, not java, there's no such thing as private/protected members just convention to put an "_" in front of things other developers probably shouldn't be messing around with.

Thanks for the insights, I appreciate it! I will take some time to go through the docs and learn more. One last question for now (I hope): after the migrations are auto-generated, do you check them into version control?

> Models are for developers and generally developers have all the permissions any way

To clarify, the reason why you would want to have read-only models is rather in complex data model cases or for performance reasons. For example, if you have a large model with 50 fields, defining a subset with 10 fields can be beneficial to performance in several queries. Other than that, I agree with you.

Yep, you do check the migrations into version control.

>For example, if you have a large model with 50 fields, defining a subset with 10 fields can be beneficial to performance in several queries. Other than that, I agree with you.

Nope, the django ORM has several helpers here. What you do in the most basic case is do something like

    MyTable.objects.filter(key__gte=12).only("field1","field2","field3")

Which will (during the initial request) only fetch the specified fields from the DB. There's an equivalent (defer) for telling it not to fetch those fields during the first query.

Of course is someone than tries to access "field4" in a template/view django will make an additional query to get that extra data.

It's a bit more complicated to make a nice chainable filter so you can call `MyTable.objects.only_foo_rows()`, you'd need to define a method on a custom object "manager", but that still wouldn't generally be handled at the model level.

There's also things like select_related, prefetch_related, and other tools to make the ORM layer more performant. Of course for really exotic things like tree-based data structures you can also perform raw SQL queries on an ORM object with `MyTable.object.raw("SQL goes here")` and have it still return ORM objects. Please don't do that though.

For a long time I didn't realize why people were so down on ORMs until I tried using a non-django ORM, it really does set the bar.

Ack on migrations, tks!

---

For completeness, you can do the same in Ecto:

    from t in MyTable, where: t.key >= 12, select: ["field1", "field2", "field3"]

However, that will still allocate a "MyTable" struct in Ecto. And if that struct is large (say 50 fields), slots for it are allocated (but none of the actual data on the fields).

I am not familiar with Python Object Model enough but, if your Python example still allocates an object with slots for all 47 other fields besides the three selected, then you may still find yourself under circumstances you would rather have a "slim" version of the model and reduce memory allocation. This may be needed when you need to load dozens of thousands of entries into memory for processing or similar.

I am not saying this is a must have but I am just trying to clarify what my initial comment was about. :)

>I am not familiar with Python Object Model enough but, if your Python example still allocates an object with slots for all 47 other fields besides the three selected, then you may still find yourself under circumstances you would rather have a "slim" version of the model and reduce memory allocation.

Honestly at that point probably don't be using python. ORM objects don't use slots, they use a hashmap to look up everything every time, and probably that hashmap points to a method that points to another hashmap that points to some kind of data structure.

If you're worried about that than django probably is not for you. That being said it scales out very nicely, you probably are not loading dozens of thousands of entries in to memory at once but are instead using a celery distributed task queue. If you did need to load that much in to memory for some reason (presumably some kind of data science?) You can use the `in_bulk("field1","field2","etc")` query option to return raw data as dictionaries/lists instead of ORM objects. You don't get the benefit of ORM methods but you're not instantiating an ORM object for every row and probably for that kind of thing you're better off taking a more functional approach any way.

Basically if you're worried about how much memory your data is going to take up you probably shouldn't be using django, hacking __slots__ or other memory efficient stuff into the django ORM is going to be tricky. In that case ignore the ORM and just construct your own objects directly or use something better suited for that.

That’s great, thanks for taking the time! Appreciated.

> I am not familiar with Python Object Model enough but, if your Python example still allocates an object with slots for all 47 other fields besides the three selected, then you may still find yourself under circumstances you would rather have a "slim" version of the model and reduce memory allocation. This may be needed when you need to load dozens of thousands of entries into memory for processing or similar.

Alternatively you can also tell the ORM to return a dictionary rather than a model instance: https://docs.djangoproject.com/en/4.1/ref/models/querysets/#...

For example, you may get a list of dictionaries with only "field1", "field2" and "field3" from a model like this:

  SomeModel.objects.filter(...).values("field1", "field2", "field3")

  [{"field1": "foo", "field2": "bar", "field3": "baz"}]

Please notice that nothing stops you from defining a non managed (ie Django migrations won't mess with it) model pointing to the same table and having declared only three of the fields of the original model.

That is more elegant than using a proxy model

The Django docs also mention that https://docs.djangoproject.com/en/4.1/topics/db/models/#diff...

> If you are mirroring an existing model or database table and don’t want all the original database table columns, use Meta.managed=False.

> For a long time I didn't realize why people were so down on ORMs until I tried using a non-django ORM, it really does set the bar.

This. Very much this.

I have given up on one finding such. The closest and nicest one that comes to mind is SQLAlchemy. The rest of languages or frameworks have indeed nothing compared to Django ORM and SQLAlchemy. It's not even fair to compare them.

My experience is also exactly like this. I have a project in java/spring and it uses hibernate as an ORM. I can't think of how much better is the Django ORM than hibernate. When people express hate for ORMs I understand where this hates comes from.

If I ever need to do another Java project I prefer to use raw jdbc sql than using hibernate.

Peewee ORM can also be nice if you need support for more esoteric DB features (sqlite user-defined table functions?). No migrations though.

Just wanted to say thank you for all the great work you've been doing. I'm a recent Elixir/Phoenix convert coming from a couple of years of full time Django and it makes me so happy to see you being interested in the way Django handles models. Imo it's one of the last things I'm struggling leaving behind (together with its tight integration of the Django admin, which you get for free).

I haven't looked close into Ash but to me it seems their approach is interesting and offers a solution, not the way Django does it but it is a solution, although a bit hard to wrap your head around at first. Django models are more simple and straightforward.

Yes. Migrations are checked into the source control. They're still the source of truth of the database changes.

Regarding read only models pointing to a tabel. It'd possible to use proxy models or even non managed models where with some mix of Managers it can provide what you're looking for.

For getting a subset of fields from a database tabel, Django provides ".only()" on the QuerySet which you can use to list all the fields explicitly and only them will be retrieved. It can even span to foreign field relations as well.

Diango ORM is indeed powerful and vrry flexible once you learn it. A novice developer can use it quickly and won't get in the way. An advanced developer can do very complex sfuff with it without touching any raw sql. Still, it's possible to write custom sql commands directly.

You can use `managed = False` and `db_table` to have another model pointing to the same table. `managed = False` will stop Django automatically creating schema migrations.

eg.

```

class ReadOnlyUser(models.Model):

    email = models.CharField(max_length=255)

    class Meta:
        db_table = "users"
        managed = False

```

> I mean you can using proxy models but that's not really a thing with django.

I think this is over-opinionated. You can use Proxy models, and this is a thing in Django. It’s not “Django 101” sure, but it’s not discouraged.

One reason you might want proxy models is if you want to only fetch certain columns for a specific type of usage. Another common use for proxy models is as a hack to get multiple Django Admin pages for the same model (again, perhaps pulling different fields or offering different views).

> Also, don’t you run into scenarios in Django where the automatic migration is not enough and you need to provide custom commands? In such cases, how do you provide them?

Data migrations are the obvious case where you need to write migration code manually. It comes up in schema migrations too on big projects. It’s easy to modify the auto-generated migration; it’s just Python that gets run by the migration tool:

https://docs.djangoproject.com/en/4.1/topics/migrations/#dat...

I don't think this is something that will come down to technical merits.

Unfortunately it is a popularity contest, where entrenched network effects are working against Elixir.

As a Python programmer, I truly believe Elixir/Phoenix is the best stack out there from a technical perspective.

However the incredible breadth of libs and resources ... and most importantly the mind share (both available devs, but more importantly available jobs for seniors who commit to Elixir) means that it is still just not a competitive choice.

Almost all commercial software dev is not about technical excellence, but rather about applying the available tech to a particular business domain. And there Python (and .Net, Java, PHP, Ruby ... even Go) are so far ahead that sadly Elixir doesn't look like it will make it.

I don't know that anything can be done about this. For all that I think Elixir is better, I, like most devs, am not about to sacrifice (or even impair) my career for the cause. Personal remuneration wins out.

That does seem like the same arguments that are made prior to just about anything becoming mainstream though. I'm not saying it means nothing that python has so much mindshare, but the same could have been said to pythons predecessors as well. You're right about commercialization not being about technical excellence, in the short term. In the medium and long term, it's the advantages of recent technical excellence that drives a lot of commercialization.

It's also the same arguments that are made prior to things falling by the wayside.

Not that I think Elixir will ever fall away, just that it won't go fully mainstream.

Very true, maybe it's not a strong predictor either way.

Reply on news.ycombinator.com