speedplane

> A business that invests in building a machine that reads data, produced by a 3rd party machine, using format intended for lay humans to read, is not investing in the right tech IMO.

Building machines that understand formats that are understood by humans is exactly what we should be doing. People should read, write, and process information in a format that is comfortable and optimized to them. Machines should bend to us, we should not bend to them.

If businesses only dealt with machine readable formats, everyone's computer would still be using the command line.

And there's real condescension in your post:

> Small-mediums should be looking to consolidate buying through a few good suppliers and working with them directly to automate process

You're saying that businesses need to change their business to accommodate data formats, but it should be the other way around.

> So, in order to extract data from them, one needs not only good PDF parser but an OCR engine too.

You can go further. Invoices often contain block sections of text with important terms of the invoice, such as shipping time information, insurance, warranties, etc. To build something that works universally, you also need very good natural language processing.

> this doesn't seem like a strong argument because this is just basic market competition. Even though a law firm being more efficient would lose hourly billing they would quickly grow more clients by providing a cheaper service. The lawfirm itself has no particular incentive to operate inefficiently.

As a factual matter, you are quite wrong. Law firms rarely get reprimanded for over-billing, and in fact, the opposite can be true.

When a lawyer bills 2X more than a competitor, they can say that they just did a 2X better job, were more thorough, etc. Lawsuits are rarely repetitive, as the facts and legal principles applied vary widely, so it's very difficult to determine what's the correct amount of time to spend on something. Worse, firms that develop reputations for big bills, can even attract big bet-the-company cases where clients can justify paying top-dollar.

The economic principal behind this is that legal services are not a typical "commodity", so standard market principles are not as effective. If you can turn a legal service into a commodity (such as LegalZoom has some parts of the law), you can be much more effective.

> In the micro, aging is a disease just like any other.

The difference is that the word "disease" is something that at least theoretically can be cured. Aging does not have that implication. It follows then that if you say "Aging is a Disease", you're effectively saying that immortality is theoretically achievable. That's what makes the claim so uncomfortable.

We just need more unions. They are historically the only thing that actually worked to level the playing field between employee and employer.

> I think Facebook is known to have more interesting jobs.

If influencing democratic elections is your idea of interesting, sure.

Python 2 Community: We are in hell, we have to stop working on everything to upgrade to Python 3, there is no straightforward way to upgrade, many of our python 2 libraries haven't been updated, and there are tons of little bugs that are hard to fix.

Python 3 Community: Look at thing cool dictionary merging thingy!

> This suggests an interesting tactic to build public support for mass surveillance.

I'm not entirely convinced that mass surveillance by government is worse than mass surveillance by huge private corporations. In democratic societies at least, the public has at least some control over the government. It has virtually none over private entities.

> junior developers ... objective is to learn as much as possible in the shortest period of time, so they can grow and get higher paying jobs. This is a perfectly legitimate, indeed desirable objective, for any individual. However, this also, often translates to a lack of interest in maintaining code.

I've seen this over and over again. It can be frustrating for a junior developer to create a large class hierarchy using arcane language features to solve a problem that can be done with a function and a for loop. The problem isn't so much that the code itself is more complex than it needs to be, it's that when they leave, someone else will have to pick it up.

> “Moving from”? That happened decades ago, before e-commerce became significant.

Yes, absolutely, since the invention of brands, at least some brands were the product rather than a signifier of quality.

However, in the last 10-20 years, largely due to globalization, high quality goods have become widely available. Before, that Nike hat may well have been much higher quality than the generic, but now they are indistinguishable. As overall product quality has improved, the only thing left to sell is the brand itself.

>>Who is your target audience (programmers, analysts, accountants)? What industries can benefit most from it?

> I'm trying to find that out.

You can start with why you use it.

> I'd be curious how you built the step from regex to ElasticSearch. My guess would be an n-gram (3-gram) index in ElasticSearch and then translating the regexes to that, but just curious if you built that custom or used something off-the-shelf. Love the site!

I'm pretty sure Elasticsearch supports regex search, it's just that it's horrendously slow and can blow up the system.

> No, marketing as driver will cause massive feature creep.

Marketing is about exposing your product to your target customers. It's relatively orthogonal to product features.

> the visit to download ratio is low compared to my other 2 commercial products. Not sure why. I'm pretty sure the traffic is targeted, I am solving a real problem and the website explains what it does. Any feedback would be welcome!

Who is your target audience (programmers, analysts, accountants)? What industries can benefit most from it? Your homepage is all about features and not about users or their problems.

> What is an example where you missed mathematical knowledge to do your job? Honestly curious, I never needed any math during programming for my job.

Basic arithmetic comes up all the time: pro-rating a monthly plan, figuring out how much to scale up or down a system in response to changes in data, figuring out when your system will run out of memory/disk.

Statistics and probability are also pretty common. I'm often calculating standard deviations and finding expected values of non uniform random variables. For example, how fast does a queue have to be to handle 1 second tasks 90% of the time, but 30 second tasks 10% of the time?

Derivatives comes with many graphics tasks, such as 3D graphics or animations.

> For heaps of this size you can use Shenandoah GC and you get pauses well below 100ms. Your post is a perfect example of the type of 'historical reasons' (or historic FUD) the GP talks about.

This GC is relatively new, and Elasticsearch doesn't support it (https://discuss.elastic.co/t/support-for-shenandoah-gc/16237...)

It may be a potential solution, but it's hardly a proven one. The concern is not FUD, it's legitimate.

> I think brand matters more than ever.

I'd go further and say that branding is now often the product itself.

Nike can sell a hat for 20X more than a generic hat when it has their swoosh on it. Perfume is basically a generic chemical wrapped in a huge amount of marketing. Apple makes excellent products, but there's no question that at least part of the premium you pay is not for better quality, but for the brand itself.

While the value of a brand often revolves around vanity and advertising, there are often other factors that are also not directly linked to product quality: such as "fair trade" coffee, buy-one-donate-one shoes, or humanely raised animals for food.

We're moving from a pure "consumer society" into something a bit different (but definitely still consumption based), where consumption is not just for the direct benefit of the consumer, but is also used to signal deeply held values.

> Why kubertetes vs docker?

Docker is for building a computer program to run on a single computer.

Kubernetes is for building a cluster, often across many computers with many programs.

They work together.

> I don't know what the actual market shares are, but when looking at open positions around the US from coast to coast, it seems to have won.

React is currently in it's honey-moon phase, but many of its warts are starting to show. It probably has another year or two before peaking, but doubtful that such a large heavy-handed library will remain dominant forever.

> Kubernetes is the thing everyone will be moving away from in the next 5 years because it was too complicated for their purposes.

Kubernetes solves a huge variety of problems with a single platform. In five years, Kubernetes will still be around, but there will be layers built on top of it that simplify common use-cases.

> 99% of the time you'll want to use query parameters, but rejecting queries which don't use them isn't a solution either, as you do need to avoid them sometimes, unfortunately

This is one of the problems that I have with most SQL databases: they try to do the magic for you and make it hard to see what's going on under the hood. Databases aren't magic, they are just indexes, and there are only so many ways of organizing them. Providing a bit more transparency into how the data is stored and retrieved allow you to write efficient queries without the guesswork.

> I certainly feel like the bourgeoisie when I use Rust or Haskell.

I feel like a peasant sharecropper when I use Java.

> Is there any notes on the business model, what worked, what didn’t, etc?

I'm skeptical of most of these post-mortems. They are generally written by the founders themselves, and therefore have a built-in bias to shift blame to external forces rather than unforced errors or a lack of understanding the product/market fit. Further, it's likely the founders probably don't know the true reasons why they failed because if they did, they could probably fix them.

That said, they do make for interesting reads, especially when read side-by-side with a similar company that did well.

> This is why hiring in software is broken. A bunch of people that don't actually want to be stewards of the craft.

I've hired a few developers and have found you can turn a inexperienced but relatively bright developer into a competent one with sufficient time and training. In my experience, this usually takes at least one year, and roughly one or two months of a manager/mentor's time. An experienced super-star can become productive within a month, requires just a few days of the manager's time, and can easily be >3X more productive than the merely competent engineer.

The problem is that it's almost impossible to identify most super-stars, and the obvious super-stars command super-star salaries (although generally less than the 3X productivity gain).

Creating a company that relies on training competent programmers may be less efficient than one that seeks out super-stars, but it's still a proven and workable model, with added benefits of higher employee loyalty and work-force stability.

> The serious point is: why all the software engineering effort for just another e-commerce website, instead of focusing on sales since day 1?

And as for their decision to open source it, why focus solely on the code that they're open sourcing, rather than the ongoing support and development they will be providing. Creating a successful open source project often comes with many of the same challenges in creating a successful company: you need to promote it, support the product and the community, and make a convincing argument that it's better than the competition.

> People often make a point that writing tests takes time, but I'm pretty sure it slows you down in the beginning of work and speeds you up quite a bit when you amount some volume, and further you go, more you benefit from having a decent coverage.

I certainly hope that quality testing is more than just your preference or even a good practice: it's now a central part of software development. At this point, we know enough about the benefits of good testing to say that it's practically malpractice to not have them. Good tests may not be a strong enough signal to conclude that a project is good quality, but lack of tests is a clear sign that a project is low quality.

> For ES and solr gurus here what is the recommended max size for documents if you want to index lot of office documents?

I run a large ES index covering 80TB of data, and regularly index documents as large as 20MB. At that size, the bottleneck is often transferring that much data over a network to the ES cluster. You need to make sure your HTTP client can handle it, and the network has enough bandwidth. Elasticsearch itself is not really the bottleneck.

> Communities can better organize and work towards becoming self sufficient and independent of governments. Grow food. Create new sources of water. Help the poor.

A well organized community that provides basic resources is essentially a government. You are basically suggesting to replace government with government.

> Does [Docker] make some things more difficult as this author says? At first, yes. As time moves forward you'll end up with many nock on effects that you can't get any other way.

I agree that docker provides many efficiency improvements, especially with respect to testing, reproducing bugs, and deployment.

But what about pure development of new features? In a bare-metal development environment, you just compile and run or launch a debugger. With docker, you need to compile, build a docker image, run the docker image, then launch your app/debugger. Would love to hear solutions on how to eliminate/ameliorate these additional steps.

> Docker gets developers and production machines on the same page. ... I would never go back to bare metal development

Instead of reading this article as "docker containers are bad" (wrong), try reading it as "there are problems with docker containers that can be improved" (right).

Containers fix a large number of issues, and many of these fixes directly benefit developers. But the article is right to point out inefficiencies that docker introduces in developer workflows. Docker may be taking developers and ops three steps forward, but for some, it's also one step backwards.