> I agree it's better to ride the trend. :)
I think you haven't looked at a 6 year graph if you think the 'trend' is only ever going up. The point of my post was to highlight the risk of (another) crash.
95 karma · joined September 20, 2016
I think you haven't looked at a 6 year graph if you think the 'trend' is only ever going up. The point of my post was to highlight the risk of (another) crash.
- PG only added checksums in 9.3
- his disks or RAID controller could be shit or he could be using some 'very cool' filesystem
- Some people turn off PG settings for performance that really shouldn't be turned off
- Some people think ECC ram isn't worth the money
- or overclock their CPU on anything other than a gaming machine
Usually it's shit hardware
I'll give you one tip.
Look at a graph of bitcoin, litecoin etc. that extends past the last bubble and collapse and ask yourself if you are willing to eat those kinds of losses if they occur again.
https://99bitcoins.com/wp-content/uploads/2014/02/bitcoin-ma...
You're totally wrong.
The easiest way to demonstrate why is for you to set up a script to randomly write zeros/junk in any amount, at any time, anywhere over one of the block devices being used by ZFS, all day every day.
[Assuming you're using one of the available forms of redundancy i.e. multiple copies, ZRAID1/2, or mirroring etc.]
Sit back and watch ZFS giving no fucks at all as it repairs all the damage passively.
You can even introduce such damage in moderate quantities across all of the block devices used by ZFS. Again, you'll see a goddamn incredible amount of self-healing going on and accurate reporting about where it's unable to recover files due to the damage across multiple volumes being too extensive.
It's unlikely that even in this extreme instance of willful massive harm to the disks you'll see the filesystem being damaged because a) filesystem metadata is checksummed too b) the metadata blocks are automatically stored twice in different places c) you also have the redundancy of multiple devices e.g. mirroring/zraid.
Try it, prove me wrong.
Unless most journaled filesystems, ZFS:
- allows a separate high-performance device to be used for the log. This is important because the cost of journalling can be high when lots of fsyncs are being used to ensure integrity (i.e. try running a write performance test on a database like postgresql using ext4 with and without journalling, you'll see a difference).
- the filesystem log can be mirrored physically, to protect against the risk of log device failure [which would endanger writes in flight].
Other similarities/differences:
In a journalling FS, you need to take the filesystem offline and check the journal. In ZFS, there is continual passive checking of file data and metadata at time of access, as well as the option for an online 'scrub' that is similar to the fsck of a journalled filesystem without requiring dismounting of the filesystem.
While copy-on-write by itself may not be necessarily strictly superior to journalling, ZFS is strictly superior to either.
Here's a quote:
- “ZFS has CRCs for data integrity
A certain category of people are terrified of the techno-bogeyman named “bit rot.” These people think that a movie file not playing back or a picture getting mangled is caused by data on hard drives “rotting” over time without any warning. The magical remedy they use to combat this today is the holy CRC, or “cyclic redundancy check.” It’s a certain family of hash algorithms that produce a magic number that will always be the same if the data used to generate it is the same every time.
This is, by far, the number one pain in the ass statement out of the classic ZFS fanboy’s mouth..."
Meanwhile in reality...
ZFS does not use CRCs for checksums.
It's very hard to take someone's view seriously when they are making mistakes at this level.
ZFS allows a range of checksum algorithms, including SHA256, and you can even specify per dataset the strength of checksum you want.
- "Hard drives already do it better"
No, they don't, or Oracle/Sun/OpenZFS developers wouldn't have spent time and money making it.
It makes a bit of a difference when your disk says 'whoops, sorry, CRC fail, that block's gone?' and it was holding your whole filesystem together. Or when a power surge or bad component fries the whole drive at once.
ZFS allows optional duplication of metadata or data blocks automatically; as well as multiple levels of RAID-equivalency for automatic, transparent rebuilding of data/metadata in the presence of multiple unreliable or failed devices. Hard drives... don't do that.
Even ZFS running on a single disk can automatically keep 2 (or more) copies on disk of whatever datasets you think are especially important - just check the flag. Regular hard drives don't offer that.
- What about the very unlikely scenario where several bits flip in a specific way that thwarts the hard drive’s ECC? This is the only scenario where the hard drive would lose data silently, therefore it’s also the only bit rot scenario that ZFS CRCs can help with.
Well, that and entire disk failures.
And power failures leading to inconsistency on the drive.
And cable faults leading to the wrong data being sent to the drive to be written.
And drive firmware bugs.
And faulty cache memory or faulty controllers on the hard drive.
And poorly connected drives with intermittent glitches / timeouts in communication.
You get the idea.
I could also point out that ZFS allows you to backup quickly and precisely (via snapshots, and incremental snapshot diffs).
It allows you to detect errors as they appear (via scrubs) rather than find out years later when your photos are filled with vomit coloured blocks.
It also tells you every time it opens a file if it has found an error, and corrected it in the background for you - thank god! This 'passive warning' feature alone lets you quickly realise you have a bad disk or cable so you can do something about it. Consider the same situation with a hard drive over a period of years...
ZFS is a copy-on-write filesystem, so if something naughty happens like a power-cut during an update to a file, your original data is still there. Unlike a hard disk (or RAID).
It's trivial to set up automatic snapshots, which as well as allowing known-point-in-time recovery, are an exceptionally effective way to prevent viruses, user errors etc from wrecking your data. You can always wind back the clock.
Where is the author losing his data (that he knows of, and in his very limited experience...): All of my data loss tends to come from poorly typed ‘rm’ commands. ... so, exactly the kind of situation that ZFS snapshots allow instant, certain, trouble-free recovery from in the space of seconds? [either by rolling back the filesystem, or by conveniently 'dipping into' past snapshots as though they were present-day directories as needed]
Anyway I do hope Mr/Ms nctritech learns to read the beginner's guide for technologies they critique in future, maybe even try them once or twice, before they write their critique.
What next?
"Why even use C? Everything you can do in C, you can do in PHP anyway!"
"I am implementing a database for a bank.
I open a transaction in a special active_customers table when the customer first joins the bank, and commit the transaction when they close their account or die. In this way, I use postgres transactions to keep track of which customers are active customers.
I am having problems with long-living customers forcing us to wait up to 120 years between autovacuums of the DB."
Clearly, that's not going to be a good way to do things.
Rossers comment (https://news.ycombinator.com/reply?id=14731214&goto=threads%...) is very helpful.
The real problem in my understanding was that I wasn't seeing that the article author REALLY IS using long-running PG transactions to as a way to enforce external queue items running to completion (or not). I know they say it in the article, indeed it's the point of the article, but it seems so strange.
Hence he's getting these long-running transactions in PG in the first place. The phrase 'sledgehammer to crack a marshmallow' comes to mind.
At 50 items/second I guess I'm totally baffled over why you wouldn't simply use e.g. an exclusive table lock each time you connect to the DB to add/take/remove a task from a task table, with a timestamp to allow aborting and rescheduling of tasks.
I just ran a test to check the performance of exclusive table locking for this purpose, and despite the slowness of having everything dumping to the console while running, and the slowness of setting up a completely new psql session for each connection (i.e. no pgpool etc), using a BASH script, and running on a crappy 6-year-old mac, I got over 100TPS.
1. I am curious if the author tried setting the autovacuum more appropriately for a table that is in continuous use at 50 new rows/second.
This article (http://okigiveup.net/postgresql-vacuuming-an-introduction-fo...) suggests:
ALTER TABLE person SET (autovacuum_vacuum_scale_factor = 0.0); ALTER TABLE person SET (autovacuum_vacuum_threshold = 4000);
This should result in entries being removed from the index and table regularly.
2. If I understand correctly, it seems the other part of the problem is that there are lots of rows to be individually tested to see if they can be locked, based on the way the author has implemented their queue. While lock attempts are being made on a set of rows, they aren't being deleted, since they're 'in use'. This non-deletion causes things to get out of control.
As far as I know, 9.5's SKIP LOCKED is meant to help address that (https://blog.2ndquadrant.com/what-is-select-skip-locked-for-...). Though it appears to be doing something kinda similar under the hood.
If the author of the original article (@ brandur.org) is following this discussion I would be grateful if they could evaluate these two options in the context of their test implementation.
However, I do not know what the situation is in the UK/EU at present, and I do not know what it was 20 years ago (when my brother was working there).
Some - many, most? - of the people they will be questioning are completely innocent of a crime.
Imagine you are an innocent bystander in some political event and a group of policemen are holding you down while one of them tears your eyelids open with his fingers and forces your phone against your eye.
Does that sound reasonable to you?
And who gets to make this decision, some policeman, on the spot, in a busy crowd, on a bad day when he's been dealing with rioters? For any reason he likes?
This is just opening up a new avenue for police brutality and physical intimidation of minorities.
Look at stop and search in the US and how it is used (specifically, who it is used against), and it is far less invasive than this.
I think policemen should be standing up to oppose this and require the use of a court warrant, as is the current standard. I wonder how many will.
Some days he would work all day and see people put money in a machine and not win anything much (e.g. jackpot never came up). Believing that the machine had a payback rate that would balance out over a few hours/days, he would deliberately put a bit of money in at the end of the 'no winner days', and would often win enough that it made it worthwhile overall.
Anecdotal, but it seems consistent with your understanding and is a source of an edge.
The majority of academics I have talked to about this in real life (let's say... 30-40 out of 50) thought we sorely needed open access in terms of:
- The public pays for science, they should be able to see the result
- Lots of out-of-work and between-jobs academics, scientists-turned-entrepreneur, scientists in small companies etc. that can't access a research library
- It's morally wrong to have the public sector hand so much money to a few private companies like Elsevier for doing very little besides existing and rent-seeking.
- Scientists and the public in developing countries and third world countries have a very limited ability to get involved in global science and engineering when it's paywalled.
- In fields like medicine especially, paywalls are literally preventing the public from knowing how to be healthy.
Have you considered that maybe the reason why for example, "The vast majority of all published papers is read only by the reviewers." is because it's not openly and freely published?
I suspect perhaps the reason you are being downvoted (on other sites presumably) is because most people think your view is wrong or unacceptable i.e. immoral. Have you considered that possibility?
I use the PRS-350 which is an old sony model that fits comfortably into one hand and can be used one handed. I recommend it.
This isn't how causation works.
> A disbeliever in EMH should look to identify these managers and pay them some fee, rather than simply investing in the index and trying to minimize fees.
It is as much work to identify good fund managers as it is to identify good company managers. You might as well save some money if you go this route and invest in a portfolio of companies directly.
> Furthermore, a smart manager will charge fees that are equal to the alpha they generate.
Warren Buffett seems quite smart, I mean he made it to rank #1 on the world's rich list and I think he's one of the few on the top #100 that did it by investing in other companies rather than just building his own. Judging from his 40 year performance data he's generated rather more alpha than any other manager. He charges fees that are very close to 0.00001% for being a partner with him.
That would seem to contradict your point.
1. It stands out clearly visually in the code.
2. It is not normally a keyword, variable or string in my code.
3. I can type it quickly and comfortably using only my left hand and two fingers, while my right hand controls the cursor to navigate (or is moving to the keyboard after cursor navigation).
I usually start typing it during the interval as my right hand begins moving left over the keyboard to prepare to type the comment that will follow.
I suspect XXX and ZZZZ are not simply conventions that are picked up by exposure to other people's code, I think they represent a local maximum of convenience for these three criteria.
Try typing any other token (e.g. TODO) and see if it satisfies these criteria as well as XXX or ZZZ.
I feel you've missed my point. Most countries offer the ability to consolidate some deductions into a standard deduction.
But in Taiwan they consolidate ALL the deductions, as far as memory serves; and the way they do it isn't a fixed amount (e.g. 5%) but the sum of what the other people achieve who go the long route, plus a bonus.
So literally it's 1-page 'sign here', or a complex form.
And it's literally a statistically 'guaranteed' win if you make things easy on everyone (average + 10%).
The US doesn't offer anything like that.
Q. How much money should I pay in tax?
A. All of it.
:-)
Increased complexity doesn't mean lower taxes or higher taxes, in principle, it just means complex taxes.
In practice, it actually leads to higher taxes to pay for an army of people to check the code is being applied correctly.
In Taiwan they have a system where you can choose to do manual deductions (A), if you think that's in your interest. Or you can take (B) 'the average deduction+10%'. Then you have a 1-page declaration of your income, no deductions marked, but they calculate what the guys who use (A) get, take the average, add 10%, then knock that off your tax bill.
No false claims, no evidence to supply, nothing to check, both sides win, each tax official can manage 100x as many Bs as As.
It's a practical example of how simpler tax codes can result in lower taxes by reducing (needlessly) big government.
Furthermore, I can predict the distribution of results for a coin quite accurately over long time timeframes. Indeed being the 'understanding' of a successful gambler means being able to make predictions about what will happen with a random process over time.
Roulette for example is analogous to the coin toss; most casinos seem extremely able to predict what will happen over time quite well.
Or are you hinting at a counterargument that most of the world is a truly random process analogous to a coin throw?
In which case, your experience with reality must be terrifying. Will the sun come up today? Who knows? Will gravity work in a minute? And so on.