Schema Migrations for Django
kickstarter.com
kickstarter.com
I'm a big believer in this type of model and having you be a Django core contributor and South developer makes it a no brainer. Paying for quality open source is always a win in my book.
Godspeed!
Maybe this is the way we'll get good, solid Python3 support for Django, quickly. I'll be more than happy to fund (in part) efforts for getting work done on other Django plugins so they work with Python3. The Django book can do with a little concentrated focus too.
I've been using django since 2006. This has been a suggestion since then. Why hasn't it been implemented if it is so necessary? It doesn't require a massive core shift, it doesn't even require that much work it seems? So why hasn't it been done with so many eyes on so well documented a platform on so accessible a language with so many commercial deployments? If no one has scratched this itch then paying for someone to scratch an itch you may think you are go
Will the person be deprived of their normal source of income for exactly the time period required to do the work? How is that justified as a cost? Maybe they are a contractor who will take the time out. If that is the case how can there be expandable requirements (longer goal objectives for more money? It could be this whole thing sits very well with their day to day job, but in the western world that is unlikely. I'm sure all intentions are good but it also strikes me as a bit questionable to do these things when lots of other people have committed much more centrally to the project with little immediate remuneration.
I also have a day a week free for this, so there's no concern of this robbing income - that day a week was previously used for learning to fly, and now I'm done with that it needs to have something that pays in order to keep me going. If the choice for that is freelance or open source work, and they pay closely, you can bet I'm going for the open source work.
Heck I've got a list of about 13 projects that are roughly affiliated with my day job but because of whom i work for and the time I have free being used up for other much more frivolous things I havn't had a chance in a year.
Perhaps it is just jealousy on my part :)
I'm sure Andrew would work on this even if the kickstarter had failed - he built South without one after all :)
But having a funded project helps in many ways. It frees him from the distraction of contract work, proves that a lot of people really care about the issue, and finally is a thank-you-donation for the work on Django in the past and the future.
Stuff like this is how I contribute. I, either personally or via companies I've worked for, have done this kind of thing a lot: approach the developer and ask for our bug or feature to get pushed up the priority queue. Sometimes it's a small tip but one company gave out a $10K contract for a bunch of work.
I don't really contribute to open source, my contribution is mostly bug reports and being willing to help the real contributors track down issues.
If this continues do you risk drying up other non-compensated commits or risk contributors waiting for funding before pushing code? You might see a small decrease, but I'd be willing to bet the same dynamics that have kept the open source community moving forward will continue - you just might see more targeted, rapid progress in areas people are willing to fund and a small cottage industry emerge that makes money contributing to projects.
"Will the person be deprived of their normal source of income for exactly the time period required to do the work?"
Does it matter? It will most likely be on his free time, which for me is many times more valuable than my normal income.
This is why developing open source is going down a dangerous path: When you work for free once, people expect you to work for free always.
Additionally, could you explain how the project will not be obviated by Django's plans to implement built-in schema migration in, I think, 1.6 or 1.7?
You taken on one of the most important aspects of managing Django services, so I'm genuinely interested, but I just want to know what the thinking is behind this.
By the way, this is the first Kickstarter project I've considered funding.
In the latest version(beta?) you can actually have a column with 'default current_timestamp' and a column with 'on update current_timestamp' in the same table after years of only being able to have 1 column per table auto that way.
I don't remember the details, but I think I had to fill a column with zeros all the way down just to drop the whole column in the migration anyway or something along those lines.
It's also worth pointing out that this IS the built-in schema migration that's going in in 1.6 or 1.7.
uhmm, it's exactly this project :)
Maybe he added it in the last 7 minutes, but this is answered in the Kickstarter description.
How will the new rebase command be different from truncating the migration table and regenerating initial migrations?
You say "Schema migration with Django has had a long and complex history". Can you explain, even briefly, why migrations haven't been a part of Django's core so far? I was under the impression that it was a deliberate design choice. What made you change your mind now?
I know that I'm perhaps the odd man out, but South just doesn't work reliably on Oracle, and I really, really miss it.
Alternately, if I were to donate a running RDS instance with Oracle on it, would that help in any way? I seem to remember an issue (that affects me) reported, and effectively disregarded with a message similar to, but not necessarily that you don't have Oracle, so it wasn't likely to get fixed.
The problem really seems to be the lack of access to an Oracle instance. Unfortunately my company isn't in a position to make available an instance, but I'm sure we could contribute an on-going dollar amount towards one.
- Are there any South "Good practices" books which you would recommend? South's documentation is great, but it might be interesting to look at some real-world project examples to learn a few new tricks.
- I'm interested in particular in data migration development practices. How do other developers go about developing and testing them? I've discovered that by raising an Exception during the data migration, South will rollback the specific migration, allowing me to iteratively test/debug the migration until it seems bug-free and ready. Are there any other ways to do it (would running the data migration with the "fake" option allow me full read access to the database, but would rollback / skip any writes?).
Thank you for your time!
There's no books or anything, alas, nor is there a very good corpus of material on this field - there's a few good talks out there, but I've not seen a great deal of written material.
Perhaps I need to write a book as well? Not sure I need another Kickstarter, though!
I'm curious about the best practices for data-migrations as well; I've seen a few places[2][3] suggest that wrapping your fixture loading "./manage.py loaddata <fixture>" in datamigrations is better than leaving it up to initial_data.{yaml,json}
Edit: here's the one I was thinking of: South common pitfalls: http://andrewingram.net/2012/dec/common-pitfalls-django-sout...
[1] https://gist.github.com/sjl/4438002
[2] http://djangosnippets.org/snippets/2897/
[3] http://stackoverflow.com/questions/5472925/django-loading-da...
Done!
Oh, and thanks for the email support now and then :-)
https://github.com/pjungwir/db_leftovers
I'll bet one natural result of it entering core will be better shared understanding of best practices & foibles of different backends.
Managing migrations on increasingly large tables is stressful for me primarily due to my lack of knowledge, moreso than fragility in any piece of the puzzle.
In any case, I've thrown in my pledge - best of luck!
(If anyone has any good recommended reading for DBA-type knowledge in a devops world, I'd love an Amazon link you think is worthwhile.)
large like lots of fields or large like millions of rows? if the latter, than migrations are a PITA and you'll often find inserting a select into a recreated new table is faster than altering the existing one. but that's probably not something a migration tool like this should/could support without lots of your work anyway.
(It may well just be my environment, but it is fairly standard RDS on AWS.)
Percona has pt-online-schema-change [1] but I have no experience with it or if it works on RDS.
There are other solutions. The most simple is doing the table rewrite yourself (but risking losing data). The more complex is to have a buffer/queue that can hold requests until the migration completes but this only works in certain specialized situations.
I'd love to find a better solution to migrations in MySQL. I've had migrations take 15 minutes on a relatively small table (2M rows). I'm a little scared to try pt-online-schema-change for fear of corrupting the db.
I would switch to PostgreSQL (nothing but love for that database from me) but the ORM I'm using has issues with identifier quoting and I can't use my "user" table with it.
1: http://www.percona.com/doc/percona-toolkit/2.1/pt-online-sch...
One of my co-workers used to pound on PGSql pretty hard in a very high traffic environment and has nothing but good things to say, but I'm not crazy about the idea of having to learn all of PGSql's foibles like we have MySQL.
My impression is that MySQL 5.6.x (we're on 5.5.x) should help with the schema migration pain, but I don't have any hard evidence for that claim.
My hope is that MYSQL 5.6.x will remove a lot of the pain, but I don't have a
Thanks for working on this, it makes me much more comfortable with my Python 3.3 decision knowing migrations will have first class support in Django. I just sent in my contribution.
We found that having this functionality as part of the core is something that makes a framework feel significantly more agile. Virtually nobody gets their schema exactly right the first time. Asking developers to to learn about and use 3rd party software (if it even exists) adds a very real friction to migrations. Altering your data model from time to time is a natural operation that occurs during application development and should not seem out of place.
I'm very excited for this, thanks Andrew!
We use Django and South in our projects, and really appreciate the work you have done. I just contributed, and am excited about this project.
My one suggestion/request: Please make it an emphasis to create awesome documentation. I am familiar with South now at this point but often find myself wishing more use cases and more examples were documented. I can usually "figure it out," but great documentation saves everyone time. This seems like a great project, so make the documentation great too!
Schema migrations are not possible in general.
Yes, you can diff two schemas and generate statements to turn one into the other. Adding columns works fine this way. Removing columns works fine.
Do you really need a library to add and remove columns? Not really. But rollback, someone will say. Schema migration libraries let you undo things. Nope. You can't rollback a drop column once it is committed, that data is gone.
What happens when you run into issues in production that you didn't have in your test/development environment?
Basically, automated schema migrations don't work. However, I guess if you haven't been through it you'll just have to learn the hard way. :)
SQL gives you a very flexible environment. You never have to break existing code to push out new features, if you are just a tiny bit careful.
South certainly has it's warts, but that's why I'm happy to contribute to Andrew's project.
Oh, and rollback: Depending on what the data is, it might actually be possible to rollback. It might be an aggregate column of some sort, or moving of data from one table another. But it's mostly not superuseful, and it's not a big part of the awesomeness of South.
I have been away from the Django project for a long time, and I really like the Rails way. I have supported anyway, everyone deserves a proper migration system.
Thanks and good luck!
Out of the goal, that's 125 pounds. How many man-hours do you think it would take to set up the site, test it, secure it, etc? I can't see that ever being worth it for relatively small goals like this! And that's before you even consider that folk are probably more likely to feel comfortable donating to a kickstarter, since they already have an account etc.
https://github.com/evanmiller/ChicagoBoss/blob/master/README...
Well, almost...it's still missing one pull. Even if it's very, very basic, I'm pretty happy to have something rather than nothing, though.
http://www.mybatis.org/migrations/index.html
It is made by the Mybatis team but it not dependent on the Mybatis object mapping framework or on Java. It runs from the command line.
I guess all the right ingredients are here: someone competent providing a useful service with an achievable goal. Nicely done and good luck!
What do you plan to do with the excess money, if you raise more than the highest, £7000 amount? Donate the remainder to the Django Software Foundation, hopefully?
Can/Do you have plans to add any additional features in addition to the existing 'stretch goals'?
I'm sure everyone has their own pet wishlist for Things I wish South Did Better, and I wonder what an appropriate fora for suggestions might be.
Mine would be some sort of 'keyframe'/checkpointing, a bit like the full history reset hack[1], but with more control over exactly where you split things.
[1] http://stackoverflow.com/questions/4625712/whats-the-recomme...
I don't know whether that sentiment scales to a broader audience, but I strongly suspect it does.
- I use South which is pretty good
- I use a NoSQL db. I will lose some things, but gain a lot as well (and in my point of view, the gains are bigger)
I'm not sure 'buying features' of software is a good way to go. (I'm not saying it's bad)
Edit: Why the hate? It certainly is better to have an integrated solution, I'm not saying this proposal is bad, only that there are other alternatives.
And the point is moot, donations went over the target already, hopefully this is going to be something big!