990 karma · joined February 9, 2010
Very powerful and scalable parallel SQL-based RDBMS systems have been commercially available for a long time. Also, I think that scalability concerns about ACID are exaggerated: ACID doesn't reach Facebook or Amazon scale, but there's only 5,000 or so sites that are that big.
A converse question to the one they ask is, "Why aren't web developers interested in Commercial Parallel RDBMS" and I think the answer to that is that there are a generation of us who grew up using open source databases, who find the thought of using a commercial database like sticking their hand in a toilet... Even if it's a reasonably priced product like SQL Server's Web Edition.
A lot of people see mysql and pgsql as going nowhere, so there's a lot of interest in something like mongodb which has a future.
If you're lucky you'll get to mess with a lot of new technology, and even meet some really cool clients, but sooner or later you're going to meet your nightmare client who burns up the profit you made on five good clients. If you're not careful, bad clients can drive out good clients and before you know it...
The industry is particularly impacted by people, who, as you're planning to do, will do work free or cheap. This puts the larger operations that are more responsible (offer health insurance, have a variety of experienced people with various specialties, always have somebody around even when some individual takes vacations, etc.) under a lot of pressure.
You're better off working on one or two big things than working on a bunch of small things that will distract you all over the place. I wouldn't go back to making web sites for small businesses even if it meant I'd starve to death or have to make my living selling aluminum siding or something.
Some people believe in the hyperlean "Four Hour Workweek" theory where you ought to be as scarce as possible.
Some businesses are about building a community. Particularly if you're looking for customer one, two or three, that's going to be a high touch proposition.
Personally, Zappos drives me nuts. I can't see how their "high touch" approach to selling clothes online can be profitable at the prices they sell things for. Perhaps it isn't, and Amazon is a sugar daddy that's going to let them make a name for themselves as people get used to the idea of buying clothes online -- Amazon can fund them for the five years or a decade that it will take for the boutiques to go out of business.
Hardly anybody is viewing "new" queues, making links to interesting web pages, or otherwise doing the work to discover what's new and good. They make it too easy for spammers and voting rings to do their evil work.
Also there's a general "burnout" effect that happens in social media -- if a particular forum isn't all that excited in your content, you'll find that it gradually gets less and less traffic because people see your URL or your name and decide to move on to the next thing. This can be overcome with real or imaginary 'social proof' (lots 'o votes) but then you get into the fact that the populations of people who are looking at the "new" queue and looking at the front page are entirely different.
This also applies to the 'entrepreneurs' who hang around, wanting to make it big but lacking any ideas about anything useful they could do.
Looking at quite a few sites, I see results that are counterintuitive: for instance, you can make more money running ads on a Chevy forum than you can on a Cadillac forum.
You might think that Caddy owners have more money than Chevy owners, however, it turns out that Chevy owners like to trick out their cars with aftermarket parts and Caddy owners (even poor owners of 20-year old Cadillacs) like to have OEM everything... Most of the ad spend is on aftermarket parts... GM isn't going to spend money on caddy forums because the people on Caddy forums know more about GM's products and GM's product plans than the people who make the ads.
I think it's also about supply and demand. Even though people spend a lot on cars, car products and car services, my experience is that there are a lot of topics that pay better than cars. The trouble is that there are too many car sites out there already, so the spend gets diluted Every gearhead and his brother has a web site about cars.
You've got to watch out for "otaku topics" like Anime. Unfortunately, anime fans are weirdos (the kind of ~guys~ who dress up like a purple-haired princess for Halloween) who don't spend a lot of money on stuff, and particularly don't spend a lot on anime because they download it all off bittorrent long before commercial dubbers start negotiating for the rights. 2/3 of the people who like anime already have a website or a livejournal or deviantart page about it, so talk about dilution... Christ, you can't pay for the storage and bandwidth costs for your images...
many sites do things with sessions, frames, javascript and such that are effectively a "romulan cloaking device" so far as Google is concerned. Then they wonder why they don't show up in the search engines.
now, there's a whole world things to think about once you're getting crawled, but there are a lot of good sites by good people that are entirely uncrawlable -- all because of ignorance of the "unwritten standards" of the web.
Why was NLTK stuck on Python 2.4 for years? I might use Python to use NLTK, but why do I have to choose an old Python to get it to work... And what if I want to use it w/ software that needs Python 2.6?
There's really no such thing as the Python language, only "Python 2.2", "Python 2.4", "Python 3". If you want to run Python scripts, you need to have three or four different Python runtimes installed, and that's asking a lot of both the distros and of users who need to keep everything straight, and probably even hack the #! at the beginning of scripts so that they work correctly.
Customs officer? Governments will be facing increasing deficits in the future and will be looking for anything they can cut.
Manager in a bank? Automation is eating the jobs of bank tellers, and will also eat the jobs of people who manage bank tellers, and the jobs of people who manage them.
As for the $90k-salaried .NET developer, his company is about to be a victim of the recession, and the next job he gets is going to be $50k a year, no health insurance.
Pensions have disappeared for private sector workers in the past 30 years, and there's serious thinking about how to make them disappear for public sectors. And if anybody thinks they're going to retire comfortably on a defined-contribution plan, they've got a hole in their head.
Somebody who could do very well in an organization that manages tasks, deadlines, architectures and conflicts well could be an NNPP or a ZNPP in an organization that doesn't.
I usually ask this question at least once a week, when the subject of the deadline comes up at the meeting.
There really are some things where, if you don't do something by a certain time, you might as well not do it at all (say, a grant application.) On the other hand, I was involved in a project that ran three months late, but somehow we managed to have meetings with the client where we always agreed we were a week behind schedule, but that everything was cool. The customer was very satisfied by that project, however, my continued involvement was expensive for my firm.
I figured whatever benefit I got from Python's other 'good' features, I'd lose back with that kind of BS, so I quit programming Python.
For example, in ASP.NET, I've had a whole team of developers swear up and down that there are no static variables in a particular segment of code that could be the subject of a race condition. Sure enough, I go in there and find one that's hidden away in a very obscure location.
Java has the same problem, as does any environment that exposes threads to the language. Sure, in theory you could do some very nice things by sharing data structures between requests, but practically the whole programming style of PHP encourages people to avoid horrible horrible problems with concurrency, and there are no language features, even in the laboratory that practically make those problems go away.
To take another example, the urldecode function in CPAN has been wrong for 15 years. If you want to write a urldecode that works right, you've got to write your own (possibly using the built-in for a base). Similarly, the URI object in .NET sometimes does incorrect transformations on URIs, and you need to use string URIs if you want to make interoperable semweb systems... PHP has the lowest fraction of 'things that say they work but really don't work' of any web programming languages, and that's the reason why it is so successful. Other languages are "blub" because they don't recognize that it's even possible for things to be this easy.
It's true that everything gets shoveled into one big namespace, but PHP lets you get a lot done with very little bull. There's a lot of crappy PHP out there, but there's a lot of crappy Java, Python, Ruby and C# too. If you approach PHP with some discipline I think you'll find it's an environment that never kills you with a showstopper flaw.
Doug Lenat was able to run a very good business (hiring about 50 people for 10+ years) running about 50% off government contracts, about 50% off sales to big companies that could use better KB management tools that his competitors had.
Cyc didn't need to change the world in order to succeed, so it didn't change the world.
That said, the fundamental trouble in NLP is the lack of common-sense knowledge. The proper word-sensing of pen in "the pig is in the pen" vs "the ink is in the pen" is a matter of semantics, not syntax. You can do Noam Chomsky stuff until you're blue in the face and it will get you nowhere... But then some stupid Markov Chain comes along that conflates syntax and semantics and beats it.
I'm amused at the general supposition that the FGCP was a failure. Yes, it failed to produce specific hardware that was commercially competitive, but the basic program of "applying parallelism to search and database tasks" behind it was the basis for Deep Blue, Google's web search and the infrastructure behind many a large scale system.