Hacker News Reader

Comment by mylons | Hacker News Reader

What limitations do you run into with the flask/python setup? (I’m not asking what ARE the limitations, I’m asking which ones specifically you run into)

View on HN

For large data sets, it sometimes spends all day in GC while just loading the data into memory (this is actually more of a problem in Ruby than in Python, but it still exists here).

It's also not very efficient about using memory, so it can be easier to get a dataset to fit in RAM in Go vs. Python.

Python also makes it hard enough to make use of multiple cores that it becomes easier to just use Go.

The same goes for making it fast. It can be done in Python, but I tend to be happy with the performance of my Go code out of the box.

I’m interested in learning more about Ruby’s garbage collection issues. Could you recommend any resources on this? Or any tools in the Ruby language that would allow me to test this myself?

https://www.joyfulbikeshedding.com/blog/2019-03-14-what-caus... is a pretty interesting resource about this.

Have you ever tried Pypy? I know when I had issues with GC and some memory usage (specifically dealing with large datasets as well, I was parsing large MySQL query results -- many millions of rows) Pypy didn't really drop memory usage much, but it did give me an almost 10x speedup. Enough that I went "oh, guess I don't need to use a different language."

I had a python server written in Python with greenlets to hat I switched to PyPy and the load it could handle went up maybe 4x. (5k per second to 20k per second. I took two weeks to switch to go and it went up another 10x to 100k per second and had less bugs (in fact a parsing bug in the python caused messages greater than 64k to be dropped; the Go handled partial reads properly and starting sending 2M msgs (to Kafka). The subscribers started getting the long messages and barfing making the system go wonky. Oddly all the giant messages were corrupt data anyways.

Code size was about the same due to stuff being implemented in Python manually being handled by Go runtime.

Reply on news.ycombinator.com