2. That is a good point. I'm hoping TimescaleDB's compression helps here but yeah I don't think you'll ever get the database size below the data's original footprint.
321 karma · joined September 5, 2018
2. That is a good point. I'm hoping TimescaleDB's compression helps here but yeah I don't think you'll ever get the database size below the data's original footprint.
Plus weather data tends to be quite noisy, just think how variable the weather can be day-to-day and we're squishing 80 years of that into one plot. Also worth noting that different places may experience climate change differently. Some places may be the average temperature go up, some maybe only in the summer, so you'll have to look at averages. Some places may see more extreme summer highs, so then you can't just look at averages but the average extremes or the tail end of the temperature distribution.
So it'll be hard to discern any climate change from just a cursory glance. I'm not saying it's there, just that it requires more analysis.
I know Timescale has a blog post from 2017 claiming a 20x higher insert rate but that's for inserting into a table with an index. The general wisdom for loading huge amounts of data seems to be that you should insert into a table with no indexes then build them later though. So with no index, inserting into a hypertable seems a bit slower.
Timescale blog post: https://medium.com/timescale/timescaledb-vs-6a696248104e
You can check out the engine code on GitHub if you're interested in the details: https://github.com/project-lovelace/lovelace-engine
There's probably some low hanging fruit in configuring Docker properly.
This website is a free open-source side project and not business critical.
We never expected this much traffic and everything is hosted on a tiny DigitalOcean server including the code runner haha. If you visit a bit later it should be much faster.
I can think of some ways to speed things up which I will try.
Don't think it's an idiom (at least not one that I've seen).
It's definitely not the nicest code, especially for simpler problems that are really one-liners but we figured lots of people would just click "submit" on just the code stubs.
I'm also thinking of how we can add a "manual submission mode" that lets you use any language that you run locally.
I guess we started simple so that the barrier to entry is low, but we should have some tougher and more interesting problems soon!
It's not trivial to add support new languages since the engine app that runs user-submitted code needs to be able to run arbitrary code in your chosen language and needs a way to communicate with Python (either directly or through JSON passing).
So far we've just been adding support for languages we know and use.
But indeed there are hundreds of languages out there that would be nice to support in some way. I'm gonna think about how we can add a new "submission mode" where you're given a couple of test cases that you run manually with any language then submit your solution.
I'm not a web developer by trade or anything so I'll have to learn how to secure the Docker container from malicious code. Hopefully Docker provides some amount of protection for now...
For today it's going to be old school htop + F9 haha.
Definitely not surprised that there are many security vulnerabilities...
I'm a grad student in computational science working on this as a hobby so building and securing websites is not something I know much about.
But now that someone is actually using it, I will take security more seriously.
Once I have some time I'll open a GitHub issue and compile a list of the security vulnerabilities you listed and will learn how to fix them.
This should be a good learning opportunity!
I think the reason we did it is so that we could visualize user output/solutions. You can learn new things by looking at your solutions to various test cases!
One example would be if you submit a solution (or just the code stub) to the Exponential growth problem, then your solution gets plotted and compared with the analytic solution and the correct solution: https://projectlovelace.net/problems/exponential-growth/
Also, unlike Project Euler where every solution is a number, a lot of scientific problems have solutions that are multi-dimensional arrays or multiple objects. So in this case, copy pasting your output becomes quite messy.
It's not trivial to add support new languages since the engine app that runs user-submitted code needs to be able to run arbitrary code in your chosen language and needs a way to communicate with Python (either directly or through JSON passing).
So far we've just been adding support for languages we know and use.
But indeed there are hundreds of languages out there that would be nice to support in some way. It might be neat to add a new submission mode where you're given a couple of test cases that you run manually then submit your solution manually.
I don't use Go so might take a while unless someone decides to swoop in and add support for it!
Supporting new languages takes some effort since we need to be able to run arbitrary Go code and communicate with Python. Definitely possible but requires some familiarity with both.
So far we've just been adding languages we know and use. Hoping to learn Rust soon so that might be the next language!
Yeah maybe we went a little too easy on the earlier problems but there will be plenty of harder problems coming up hopefully!
I actually think Project Euler gets hard pretty quickly but I'm no number theory wizard haha.
We thought it would be tedious to write tons of code to make sure users don't take the easy way out for each problem and language, so we figured we might as well allow it. We can't force users to solve the problem our way.
For running Javascript and Julia, it goes something like Python objects -> JSON -> read JSON in Javascript/Julia -> run code -> output JSON -> read user output from JSON in Python.
For C, we can call C functions directly from Python with some code for dealing with different types.
Not sure if this is the best approach (it's not super fast) but we've been learning as we go. We might be due for a refactor in case the next language we want to support doesn't fit into this pattern. I'm personally excited to learn Rust and maybe add support for it.
PS: Our stack is on GitHub in case anyone is interested in having a look: https://github.com/project-lovelace
Ah I'm no web dev so must have missed that margin issue. Was able to reproduce in tablet mode. Thanks for pointing it out!