mtrn

From a developer perspective many aspects of the current system (with respect to resource usage and allocation) look like technical debt. Governments may try to make ends meet with limited budgets and come up with sometimes nonsensical or just not really thought out solutions to problems. Meanwhile the world learns about the Panama papers and related things, getting the impression, that not everybody is happy with contributing back to the community at large.

The simple minded software developers solution would be to radically cut out the cruft (complex laws and tax systems) and implement a simplified version, that would be harder to game and easier to debug.

> One problem is that people are too macho to admit they have cognitive limits.

Yes, I experienced this and it seems to be a real problem, for various reasons: if it does not look complicated, investors won't like it, won't buy it; if it is too simple to use, people will think I am dumb. If the setup is too simple, people will think I am incompetent, etc.

Simplicity is a prerequisite for reliability, not a nice-to-have.

> but the essence of science, the pursuit of better understanding, is lacking

When I read articles and books from scholars from the 19th and beginning 20th century, I'm often surprised how broad their horizons were and how cautious in their thinking. It's always a pleasure to read.

Funny, people seemed to have used the site because it was owned by academics, e.g. http://kennethcwilbur.com/whyssrn.html

> Since SSRN is owned by academics, I expect it will continue offering similar services for the foreseeable future.

BTW: Did SSRN had a official method of downloading their metadata?

Thanks, just sent an request for invitation.

Great, thanks, would love to contribute since it looks like a useful product.

Excellent, thanks!

Never heard of infinit.sh, but it looks great. The repo is almost empty, though (https://github.com/infinit/infinit).

Is the code completely open source? Could I setup infinit.sh inside my own infrastructure?

I don't believe in rewrites. But I believe in the evolution of the understanding, what a system is supposed to do and how to do it.

A customer might have a lofty web-scale vision, the first version is built with all scaling and no maintenance in mind. You are then faced with a decision: carry along the visionary nightmare and get almost no real work done, because the system itself is in the way, or rewrite the system with a fresh, minimal approach, focusing on a valuable target.

The point being: Software requirements evolve and sometimes, writing a first version of the software is the spec itself. Just as with specs, you can try to scavenge all good things from the previous version for the next one - and actually getting better over time, despite being in a rewrite mode.

I don't think so. HN frontpage HTML is about 31201 bytes at the moment. Techcrunch 249162, Google 64871. Here, less is more.

youtube-dl is great. It belongs to a class of software, that could not exists without a place like github (or any platform, that makes it easy to contribute). Many developers (here: 403) sharing the burden of keeping the scrapers up-to-date, which would be a dull and mind-numbing task for a single person or small group of people.

Other wonderful projects in this category would be homebrew[1], wiki*, OSM, ...

Would love to share experiences or collaborate (contact in profile if you are interested).

Interesting. To be honest, a static data set would be perfectly fine for a first batch processing attempt.

> that could be used to get the kind of data you need.

Cool. Would you be interested in sharing or exchanging data?

I see scientific agents on the horizon and with them a new wave of research. But access to raw data, e.g. publications, is very limited and restricted. It's a pity.

I would love to have a list of URL of all .edu domains, which contain the word "publications". It's a bit silly, but I'd love to build a open version something like Google Scholar.

I can think of many other use cases as well, where a product needs to built from larger, but very selected set of raw input or pages.

BTW: Thanks for Hackday Paris 2011! Loved that event and venue :)

I would pay for an API, that gives me access to partial quality crawl content.

It would be even better, if the web would be treated, at least for parts, as a digital library and non-profit organizations would recognize the value of access to such a resource and provide it (just as maintenance for roads or public schools).

I guess, the library authors needed to learn Go themselves as they build up the stdlib.

Turbo Pascal 5 got me into programming. I wish people would build smaller things again.

The Go standard library. At least two things coming together: A stripped down language, that explicitly aims to be readable and experienced programmers / authors.

Sticking with a MacBook Pro 13" as well. Gives you a hard limit on the length of functions that fit on a screen. Upgraded to 16G RAM, 1TB SSD + 2TB HDD (not really possible with Retina), since I work a lot with data. Would love to upgrade to 2 x 2TB SSDs some time.

Flink comes from a more database oriented background. It grew out of research project at TU Berlin[1].

I believe, Flink tried to focus on query language and optimization, as you probably would in a database settings. In contrast, Spark is sometimes described as a batch processing system, which provides a real-time experience by intelligently partitioning the work.

[1] http://stratosphere.eu/

They might not be perfect, but my uBlock says 611,492 or 8% blocked requests, which still helps.

Thanks for your insight and encouragement. I actually studied compsci and creative writing, so I have a background in both. I am still working as a software developer / "data engineer" by day, and I am starting to sketch out a small book by night.

I am reading and writing Go and Java almost daily. Java has a tendency to be written in an over-engineered way. The Go community has an inclination towards cautious abstractions.

Take interfaces. In Java you might start with them. In Go - in the best case - they emerge, when it's time for them. Java has more mature tooling, but then, I cannot remember gathering runtime insights with Java quicker than with Go pprof[1].

Java is rock solid and more and more libraries are written in lightweight and clean way. Go has more momentum and also a growing number of curious users, who explore the language in creative ways[2].

[1] Google image search for "go pprof": https://goo.gl/6l7t1b

[2] Just discovered this amazing guy recently, https://github.com/fogleman/ln, https://github.com/fogleman/nes

Reminds me of this dialog from Spectre (http://www.imdb.com/title/tt2379713/):

> Have you ever had to kill a man, Max? Have you? To pull that trigger, you have to be sure. Yes, you investigate, analyze, assess, target. And then you have to look him in the eye. And you make the call. And all the drones, bugs, cameras, transcripts, all the surveillance in the world can't tell you what to do next.

A decade ago, LyX was my gateway to LaTeX. It was really fun to use it to create beautiful documents.

At my day job I work a lot with data, from a multitude of sources. It amazes me how hard it is to build data products (you could think of it as an analog to scientific experiments) and know exactly what parts of the processing pipeline influenced what parts in the result.

On the other hand: When I hear that many scientists use Excel for complicated processing chains and are sometimes themselves hardly able to reproduce anything a few month after the paper was published, I believe we hit some new low point in scientific activity. Combined with an infrastructure focused on research impact and number of papers published, this is really a sad situation.

There is certainly some involvement of magic - read: metaclasses[1]. However, I just want to add another anecdotal data point. A few month back we had a four hour Django session with interested teenagers. Most of them had no problem picking up Django based on a prepared tutorial. Even though other frameworks might be more lean and slim, Django has still a lot to go for it in terms of comprehensiveness and ease of start.

[1] http://stackoverflow.com/q/100003/89391

We have been using Luigi for a larger project and it works fine. Some people have a bit of a hard time understanding what it is about and why at least some software for scheduling is needed. I find the notion of "make for data" useful.

After a presentation on Luigi in a Python User Group, we had a lively discussion about certain features. One issue that came up was the fact, that downstream tasks are not necessarily recomputed, once you change something in the code. For that to happen, you would have to keep track of the source code as well. Similarities with Nix came up, where a change in code leads to a different ID, so all changes can be tracked.

Shameless plug: When I started using Luigi, I missed some auto-generated filename feature for task outputs, so I wrote a utility library for that (and a few other things): https://git.io/vg4D0

Why is Go packaging terrible? Because we are going back to static binaries?

I write and ship Go and Python code every day and I found that distribution is a great overall benefit - one, that some developers for some reason seem to ignore.

It's not just convenient for me, but for various other parts of a project as well: less moving parts is welcomed by ops, fast iterations help to meet the requirements. Overall a very positive impact for a reasonable price tag: larger binaries and recompilation overhead, when a critical library that is used needs a fix.