HNHacker News
TopNewBestAskShowJobs

wkornewald

1,122 karma · joined January 23, 2010

Working on a declarative, synthesizing, AI-based programming language (Ensody). Interested in PL design, AI, server automation, end-to-end encryption.

https://www.ensody.com

https://github.com/wkornewald

https://github.com/ensody

https://bitbucket.org/wkornewald

submissionscomments
wkornewald··on NoSQL Joins, or, So You're Writing an RDBMS
Thanks a lot for the kind words. :)

BTW, if you have any feature request for django-mediagenerator please let us know.

wkornewald··on NoSQL Joins, or, So You're Writing an RDBMS
Could the title please be changed to something that makes more sense? This is not an attempt at re-creating an RDBMS. It's about providing an automated and abstract mechanism for denormalization. When writing scalable systems you most likely won't get around denormalizing at least parts of your data. The problem with denormalization is that is requires lots of additional code and it puts lots of inconvenient dependencies in your code, making clean code separation difficult or impossible. The dbindexer is our attempt at solving this problem.
wkornewald··on Apple.com gets a redesign
That's what asset versioning is for. I can't believe they don't concatenate their assets. Even their versioned PNG files use Etags... With proper caching and other optimizations they could probably get their site load times down to maybe 300ms (almost instantly) instead of 1.8s and at the same time fix such caching problems.
wkornewald··on CPython to LLVM to Javascript - Python in the browser
I really don't remember the exact numbers, anymore. A few times slower. The slowdown certainly is significant enough for number-crunching code, but end-users wouldn't notice the difference for simple calculations. I haven't yet played with per-module compilation flags, but it would be possible to compile performance-critical modules with a higher optimization level. Such a solution might require a little wrapper to convert between the data types, depending on the chosen compilation options, but it's the same when talking to raw JS code.
wkornewald··on CPython to LLVM to Javascript - Python in the browser
Note that pyjs has several compilation options, one of which wraps numbers to distinguish between float and int. However, stuff like correct delegation with multiple inheritance might never work because js can only have one delegate. Currently one of the big problems in practice is that attribute access doesn't behave like getattr, which makes working with @property inconsistent. Also, __call__ doesn't work. Lots of minor details. At least they are checked in the unit tests, in case someone wants to help.
wkornewald··on Reinventing the wheel with microframeworks?
Caching is already pre-configured (in default settings) to use GAE's memcache in djangoappengine/Django-nonrel. The session app is also pre-configured to use cache+db. Django's mail API also is pre-configured to use GAE's mail API via a special backend. Even file uploads work with FileField if you use django-filetransfers. A large part of Django already works out-of-the-box and it's easy to write fully portable code.

With django-dbindexer you can even use QuerySet features which are not natively supported by App Engine. At some point django-dbindexer will even support JOINs via automatic denormalization (or via in-memory filtering where desired) and maybe also simple aggregates and scalable counters (via QuerySet.update()). This will simplify complex DB code a lot (compared to App Engine's model API) and by reusing Django's ORM as the abstraction for common NoSQL patterns you can not only move between NoSQL DBs, but also between SQL and NoSQL (at least to a certain extent; the goal is to provide a better high-level NoSQL API and only implement scalable features, not build a SQL emulation layer that wouldn't scale).

wkornewald··on Reinventing the wheel with microframeworks?
Transactions via run_in_transaction() do work (it's even mentioned in the docs).
wkornewald··on Reinventing the wheel with microframeworks?
1. You can still use transactions with Django-nonrel. Just use App Engine's run_in_transaction() function in combination with Django models. No problem.

2. The slow startup problem doesn't really matter, anymore, because App Engine provides reserved instances and warmup requests. It only matters to those who don't want to pay a few $ for reserved instances (but even without reserved instances there are ways around that ;).

wkornewald··on Oooh - I'd kill for a four day week...
Actually, I wonder at which point you guys notice that your productivity drops. Very often my productivity drops noticeably after 7 hours, already, but it can come back in the evening.

Also, it often drops after lunch, but this also depends on when we have lunch. It's significantly worse if we have lunch around 12:00 than around 13:30-14:00. Our research group loves to go at 11:30 (!!!) which destroys the whole day for me, so a few of us have just stopped eating together with them.

Update: I've added a poll: http://news.ycombinator.com/item?id=2052007

wkornewald··on Git and Mercurial branching
So hg branches should behave like completely separate clones/repos that happen to live within the same folder/URL. That would fix pretty much all issues and be completely straight-forward since it mimics the separate-clones-as-branches model originally promoted by hg, with the difference that you don't have separate folders for the clones.
wkornewald··on Prototype based vs. class based inheritance
Yeah, and that's a real shame because those developers will get a really bad impression of prototypes (unless they only use one of the class emulation libs and never learn about prototypes, of course).
wkornewald··on Prototype based vs. class based inheritance
JavaScript has the worst form of prototype inheritance I've ever seen. All the boilerplate code is not the fault of prototypes, but of JS. Also JS prototypes are insanely limited. You can't have multiple delegates and you can't change delegates at runtime. JavaScript's inheritance actually smells like some broken state between class based inheritance and prototypes. It has none of the advantages of either and combines the disadvantages of both. I'll make a longer blog post explaining how prototypes are supposed to work and what their real advantages are. You'll see that JS inheritance is a huuuge design failure and that nobody should ever mention prototypes and JavaScript in the same sentence.
wkornewald··on Happy Holidays from the App Engine team - 1.4.0 SDK released
This release solves almost all problems we had with App Engine. Users now don't have to wait for an instance to start, so the site will always be fast. Also, DeadlineExceededError now can't lead to mysteriously broken instances, anymore (thanks Guido!). And with the Channel API we can build real-time collaborative apps.

Now we only need SSL support and here we come, blasting everyone away who wastes time on maintaining his own servers. :P Well, at least until the next feature is needed, but not available, so let's hope this won't be the case. ;)

wkornewald··on Why we switched from Google App Engine to EC2
Cold start times won't matter, anymore, when App Engine 1.4 gets into production - which is soon. Then you have warmup requests and reserved instances. At that point small frameworks will lose all their appeal for non-trivial projects.
wkornewald··on Why we switched from Google App Engine to EC2
Could you please explain what you don't like about Django-nonrel? Why do you believe that webapp or tipfy are better?
wkornewald··on Django-mediagenerator
Do you really want to say that designers are incapable of adding a little string to a settings file? You could easily move the media-related settings into a separate file, so there's really no confusion, at all.

There are several reasons why assets should be configured in the settings. The most important ones: (1) This is the only way to support sandboxes like App Engine. (2) This makes advanced features like HTML5 manifest support possible, at all (you have to know all generated assets in advance). (3) Assets are pre-generated, so when deploying a new version your users don't have to wait for yuicompressor and other stuff (e.g. upload to S3) to finish.

The only advantage of the template-based approach is that it's more "familiar". However, that approach has too many limitations and disadvantages. That's why we went with settings-based asset definitions.

wkornewald··on Google App Engine is now much faster, after latest maintenance
The performance degrades too quickly for this. In two or three months we might be on our old performance level, again. Well, I seriously hope that we'll stay fast this time. Only time will tell.
wkornewald··on Google App Engine is now much faster, after latest maintenance
It's still too early to tell. This would not be the first time that speed improves after a maintenance and then it slowly gets worse every week until it looks almost like before the maintenance.
wkornewald··on Using the Cython Compiler to write fast Python code
No, it's a custom implementation of a simple compiler. It's nothing complicated. It converts Python to C++ and compiles that with nvcc. It also supports numpy arrays. It doesn't do any complex optimization steps like a full compiler. It's more like Cython, actually (with type annotations via an @gpu decorator). This allowed me to take my Python image processing code almost literally and annotate it with @gpu. The code isn't released, yet.

I originally wanted use Copperhead and got in contact with the developer a year ago, but it was too early even for "private" beta testing, so I never got access to their code. Also, my compiler is specialized on image processing, so probably Copperhead wouldn't have worked, anyway. I'm only jealous of Copperhead's type inferencer. :) But then again, I have to get finished with my thesis and a type inferencer wouldn't help with that goal. ;)

wkornewald··on Github has won
Note that it's not 5 free private repos, but 5 private users. You can have unlimited private repos. So, in a small team you'd probably never have to pay. I wonder what has led them to this decision business-wise. Was the competition from github so overwhelming that they decided to counter it with with relatively drastic measures (giving up lots of $$$)?
wkornewald··on Using the Cython Compiler to write fast Python code
Yeah, I got the runtime down from 40min to 10min. Then I implemented a Python->CUDA compiler and got it down to 30sec. :)
wkornewald··on What, more tests are always the best way to improve my product?
Test coverage is a fake feel-good metric that makes you want to increase the number and by this give you the impression of progress. Real progress is measured differently. Test as much as necessary, but not everything.
wkornewald··on Using the Cython Compiler to write fast Python code
Then you're lucky because when I optimized an algorithm for my diploma thesis the code grew significantly because my Python functions suddenly had to deal with different type combinations and also, as part of the optimization, I had to switch from my short numpy-based code to manual for-loops which combined several operations that were simple (but less efficient) numpy arithmetic before. I think it highly depends on what you want to do, whether you can still work with Python objects, and how much static typing gets in the way when trying to solve your particular problem.
wkornewald··on 12 Months with MongoDB
Not sure if this is important for you to know, but anyway: I'm one of the creators of Django-nonrel (a fork of Django which adds support for NoSQL DBs to the ORM), I've helped with the development of the MongoDB backend, I've developed a large part of the App Engine backend, and I've worked for over two years with NoSQL solutions.

As you've said, scalability is one valid reason for using a NoSQL solution. Sometimes you also have special needs for multi-datacenter replication or whatever. Sometimes a NoSQL DB can fit your problem better than a SQL DB. However, what does it look like from a development standpoint?

If you have an offline-capable app it might be easier to implement it with CouchDB.

Also, CouchDB's MapReduce views could allow you to run queries that wouldn't be possible with SQL. Note that MongoDB also has MapReduce support, but it rebuilds the whole index instead of only the parts that changed, so on a large DB this might take too long and thus be impractical. Anyway, I guess you're talking about NoSQL solutions in general.

In some cases you actually want to store very flexible schemaless data in your DB. SQL would be a huge pain, here.

It might also be easier to switch from an XML DB to e.g. MongoDB.

Migrations can sometimes also be easier with a schemaless DB. This could be important when experimenting or developing iteratively.

Anyway, most of these are all rather special use-cases and that's probably the important development-side strength of NoSQL: Having solutions for special problems.

Some people here say that working with NoSQL is more natural than with SQL. If you're comparing hand-written SQL code with hand-written code for the low-level MongoDB API then that's indeed true. However, in practice you should better use an ORM because that makes you more effective. When comparing, for example, the Django ORM with MongoEngine you'll see that the APIs look similar. It's just that SQL gives you infinitely more powerful queries and an ORM makes writing even complex queries very easy. A NoSQL DB sometimes forces you to jump through hoops to implement something that would be a two-liner in SQL+ORM. So, from this point of view NoSQL has nothing to offer for a large number of problems. Of course, some projects only use very simple queries which map nicely to NoSQL. In these cases it doesn't make any real difference whether you use SQL or NoSQL, but that still is not a real plus for NoSQL.

One other reason that once spoke for NoSQL: Google App Engine and Amazon SimpleDB were among the first solutions which offered scalable, replicated, managed hosting with an acceptably cheap pay-as-you-go model. Today we have Amazon RDS, MS SQL Azure, and soon App Engine will have SQL hosting, too, and there are several other cheap cloud hosts with SQL support. So, this reason isn't valid, anymore. It might still be part of the hype, though.

Yes, NoSQL is indeed over-hyped. Too many newbies use it for the wrong reasons (esp. on App Engine where they use it because they can save $3/month). Only if you need to scale or if your project has special needs you might find rescue in the NoSQL world.

Finally, since many of us are working on a web startup: As long as it's not clear if you'll become successful and thus have to scale you might be tempted to use SQL and develop faster in the early phase of your project. However, when looking at a complete web project the development/code overhead caused by NoSQL (vs SQL) is often not very large (well, depending on your particular project), so you might want to go with NoSQL right from the start and not have scalability woes later on. Anyway, you need to make an informed decision for your particular problem. Don't just pick NoSQL because it's hyped and also don't just pick SQL because you're wary of hype. :)

BTW, if you want to go with NoSQL and still have more complex queries like with SQL you should join our open-source project django-dbindexer. The goal is to automate denormalization and index generation such that you can use e.g. simple JOINs and aggregates with Django's ORM instead of emulating them with hand-written code. This should be useful to a lot of web startups and make it easier to use NoSQL instead of SQL right from the start and not worry about scalability. See here:

http://www.allbuttonspressed.com/projects/django-dbindexer

http://groups.google.com/group/django-non-relational

wkornewald··on When App Engine Went Wrong
If they don't use JOINs or aggregates your apps might just run unmodified. In some cases you'll need extra index definitions, either for djangoappengine or django-dbindexer, but you can keep those separate from your app, so you don't need to fork the code. BTW, I've updated the docs and added a short note that should answer your original questions.
wkornewald··on When App Engine Went Wrong
OK, thanks. Will try to remember that. :)
wkornewald··on When App Engine Went Wrong
It's based on the latest trunk code. The only changes are in the ORM layer and they're pretty small (maybe less than 100 lines). Which other questions do you want to see answered on the website?
wkornewald··on When App Engine Went Wrong
Did you try Django-nonrel? It supports a lot of Django features out of the box (even the admin). Also, it makes it a lot easier to move somewhere else. You might not be able to switch to SQL, but you can at least switch to MongoDB or some other NoSQL database.
wkornewald··on When App Engine Went Wrong
DeadlineExceededError is a really stupid problem in App Engine and the simplest fix would be to just kill the instance after such an error and start a new one. Google doesn't want to do that because it wastes resources. So, the current solution is to write Python modules which, when they're imported, will work correctly even if the execution stops in the middle of the module and then is restarted from the beginning (i.e., when a new request is received).

It's possible that with the upcoming pre-warming requests this bug will disappear in practice because your whole project gets a chance to import and initialize all modules before the first request is received. If these pre-warming requests don't have a 30s timeout your instance will always have enough time to finish the initialization in a non-broken state.

Anyway, it's your own mistake if you stay with the Django helper. The bug can be worked around. It's fixed in Django-nonrel. I still see people who try to use the helper for mission-critical projects. Don't do that. The helper is buggy. It uses monkey-patches all over the place. This makes it extremely vulnerable to bugs caused by DeadlineExceededError. Just use Django-nonrel, even if you only want to use App Engine's models.

wkornewald··on When App Engine Went Wrong
It's because a little VPS won't give you a fail-safe system. For a SaaS startup you'll need to have several machines in a redundant cluster and ideally you'll also have machines in at least two data centers. The people who use App Engine for their business all have the same dream: Build the app, click deploy, and have Google handle all the annoying server stuff. Obviously, App Engine isn't quite there, yet. It's not fast enough, it's not stable enough, and it has a limited feature set. Clearly, we're early adopters. However, App Engine is going in the right direction and once the problems are gone people will ask "Why not just use App Engine?" (or whatever other PaaS) instead of "Why not just use a VPS?".
← PreviousPage 2 of 3Next →