HNHacker News
TopNewBestAskShowJobs

PolarizedPoutin

321 karma · joined September 5, 2018

submissionscomments
PolarizedPoutin··on Loading a trillion rows of weather data into TimescaleDB
1. Yeah to me it seems very hard to saturate I/O with Postgres unless maybe you insert into an unlogged table. I guess there's quite a bit of overhead to get all the nice stability/consistency and crash-resistance.

2. That is a good point. I'm hoping TimescaleDB's compression helps here but yeah I don't think you'll ever get the database size below the data's original footprint.

PolarizedPoutin··on Loading a trillion rows of weather data into TimescaleDB
Haha very cool use! Yeah reading up on TimescaleDB vs. Clickhouse it seems like columnar storage and Clickhouse will be faster and better compress the time series data. For now I'm sticking to TimescaleDB to learn Postgres and PostGIS, but might make a TimescaleDB vs. Clickhouse comparison when I switch!
PolarizedPoutin··on Loading a trillion rows of weather data into TimescaleDB
I have not analyzed any data yet and the purpose of plotting the time series was to show an example of the data as a function of time. As others have already mentioned, the swing in Durban temperatures over the seasonal cycle is ~25°C while global temperature increases due to climate change so far are on the order of 1°C.

Plus weather data tends to be quite noisy, just think how variable the weather can be day-to-day and we're squishing 80 years of that into one plot. Also worth noting that different places may experience climate change differently. Some places may be the average temperature go up, some maybe only in the summer, so you'll have to look at averages. Some places may see more extreme summer highs, so then you can't just look at averages but the average extremes or the tail end of the temperature distribution.

So it'll be hard to discern any climate change from just a cursory glance. I'm not saying it's there, just that it requires more analysis.

PolarizedPoutin··on Loading a trillion rows of weather data into TimescaleDB
Thanks for the insight! It is true that I started with a micro-benchmark of 20k rows for slower inserts, but I also did some longer benchmarks with ~772 million rows.
PolarizedPoutin··on Loading a trillion rows of weather data into TimescaleDB
Yeah I'm curious about this too. Been meaning to ask on the Timescale forums. My only guess is that there's some small extra overhead due to hypertable chunking.

I know Timescale has a blog post from 2017 claiming a 20x higher insert rate but that's for inserting into a table with an index. The general wisdom for loading huge amounts of data seems to be that you should insert into a table with no indexes then build them later though. So with no index, inserting into a hypertable seems a bit slower.

Timescale blog post: https://medium.com/timescale/timescaledb-vs-6a696248104e

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Oh that's a neat idea. I should check out Advent of Code. Summing would probably work for a lot of problems but some problems mix strings with numbers in the output
PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
I'm seeing a "404 Not Found" when I visit (after logging into NextJournal).
PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Very cool! Didn't know there was a classical mechanics counterpart to SICM. Could inspire some new problems for the website.
PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Thanks for the suggestion! I'm all for quality of life improvements. Definitely wouldn't be hard to add I think.
PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
The code gets uploaded to the server which sends it to an "engine" sitting in a Docker container. The engine generates test cases and send them with the code to another Docker container where the code is run. Then the output from your code is sent back to the engine which checks to see how many test cases you got correct before sending all the information back to your browser.

You can check out the engine code on GitHub if you're interested in the details: https://github.com/project-lovelace/lovelace-engine

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Nice to see another Julia user! Yup right now we only have 27 problems but there will hopefully be many more in the future. Some people might even contribute some new problems.
PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Thanks for the suggestion. Thread is too deep to reply to your actual message, but I will look around to see how other "online judge" software run arbitrary code securely.

There's probably some low hanging fruit in configuring Docker properly.

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
I'm open to suggestions and advice if you have any.

This website is a free open-source side project and not business critical.

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
That sounds awesome! Definitely let us know if we can help.
PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Thank you and sorry about the slow load time.

We never expected this much traffic and everything is hosted on a tiny DigitalOcean server including the code runner haha. If you visit a bit later it should be much faster.

I can think of some ways to speed things up which I will try.

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
I agree 100%. It seems that almost any real-world science problem these days requires a computational solution/approach at some level. Taking a computational approach in parallel would be super useful.
PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Yeah the other two replies explain why we used code stubs that would at least run but not pass.

Don't think it's an idiom (at least not one that I've seen).

It's definitely not the nicest code, especially for simpler problems that are really one-liners but we figured lots of people would just click "submit" on just the code stubs.

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Haha thanks! Didn't know about codewars.com, will look into it. Might be a cool way to learn Rust syntax.
PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Java is definitely on the list of languages to support soon!

I'm also thinking of how we can add a "manual submission mode" that lets you use any language that you run locally.

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Yeah haha they're definitely simple if you have almost any education in math or science.

I guess we started simple so that the barrier to entry is low, but we should have some tougher and more interesting problems soon!

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
I'm excited to learn Rust actually so we might support it soon!

It's not trivial to add support new languages since the engine app that runs user-submitted code needs to be able to run arbitrary code in your chosen language and needs a way to communicate with Python (either directly or through JSON passing).

So far we've just been adding support for languages we know and use.

But indeed there are hundreds of languages out there that would be nice to support in some way. I'm gonna think about how we can add a new "submission mode" where you're given a couple of test cases that you run manually with any language then submit your solution.

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
There should be a limit on how long the Docker container can run code for, but it might be unnecessarily long right now.

I'm not a web developer by trade or anything so I'll have to learn how to secure the Docker container from malicious code. Hopefully Docker provides some amount of protection for now...

For today it's going to be old school htop + F9 haha.

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Thank you so much for taking a look and letting us know!

Definitely not surprised that there are many security vulnerabilities...

I'm a grad student in computational science working on this as a hobby so building and securing websites is not something I know much about.

But now that someone is actually using it, I will take security more seriously.

Once I have some time I'll open a GitHub issue and compile a list of the security vulnerabilities you listed and will learn how to fix them.

This should be a good learning opportunity!

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Right yeah it's a good question.

I think the reason we did it is so that we could visualize user output/solutions. You can learn new things by looking at your solutions to various test cases!

One example would be if you submit a solution (or just the code stub) to the Exponential growth problem, then your solution gets plotted and compared with the analytic solution and the correct solution: https://projectlovelace.net/problems/exponential-growth/

Also, unlike Project Euler where every solution is a number, a lot of scientific problems have solutions that are multi-dimensional arrays or multiple objects. So in this case, copy pasting your output becomes quite messy.

It's not trivial to add support new languages since the engine app that runs user-submitted code needs to be able to run arbitrary code in your chosen language and needs a way to communicate with Python (either directly or through JSON passing).

So far we've just been adding support for languages we know and use.

But indeed there are hundreds of languages out there that would be nice to support in some way. It might be neat to add a new submission mode where you're given a couple of test cases that you run manually then submit your solution manually.

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
That is true. It's not super easy to add new languages and I'm not familiar with Fortran (thankfully our group switched from Fortran to Julia recently!) but we already support C since it's easy to call C from Python. Perhaps Fortran support won't be super hard either.
PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Thanks for checking it out!

I don't use Go so might take a while unless someone decides to swoop in and add support for it!

Supporting new languages takes some effort since we need to be able to run arbitrary Go code and communicate with Python. Definitely possible but requires some familiarity with both.

So far we've just been adding languages we know and use. Hoping to learn Rust soon so that might be the next language!

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Thanks for checking it out! Will look into the LaTeX rendering issue.

Yeah maybe we went a little too easy on the earlier problems but there will be plenty of harder problems coming up hopefully!

I actually think Project Euler gets hard pretty quickly but I'm no number theory wizard haha.

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Yeah haha, this is somewhat intentional!

We thought it would be tedious to write tons of code to make sure users don't take the easy way out for each problem and language, so we figured we might as well allow it. We can't force users to solve the problem our way.

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Ah do we run all user-submitted code in Docker containers. The "engine" that runs the code is written in Python and we do different things for different languages.

For running Javascript and Julia, it goes something like Python objects -> JSON -> read JSON in Javascript/Julia -> run code -> output JSON -> read user output from JSON in Python.

For C, we can call C functions directly from Python with some code for dealing with different types.

Not sure if this is the best approach (it's not super fast) but we've been learning as we go. We might be due for a refactor in case the next language we want to support doesn't fit into this pattern. I'm personally excited to learn Rust and maybe add support for it.

PS: Our stack is on GitHub in case anyone is interested in having a look: https://github.com/project-lovelace

PolarizedPoutin··on Show HN: Science and math-flavored programming problems for beginners
Haha thanks for taking a look!

Ah I'm no web dev so must have missed that margin issue. Was able to reproduce in tablet mode. Thanks for pointing it out!

← PreviousPage 2 of 2