Japan HP accidentally deleted 77TB data in Kyoto U. supercomputing system
iimc.kyoto-u.ac.jp
iimc.kyoto-u.ac.jp
They do not try to blame it on complex systems or other factors.
Users lost 1 day and 1/2 of recent work (which doesn't seem to be that bad).
About file loss in Luster file system in your supercomputer system, we are 100% responsible.
We deeply apologize for causing a great deal of inconvenience due to the serious failure of the file loss.
We would like to report the background of the file disappearance, its root cause and future countermeasures as follows:
We believe that this file loss is 100% our responsibility.
We will offer compensation for users who have lost files.
[...]
Impact:
--
Target file system: /LARGE0
Deleted files: December 14, 2021 17:32 to December 16, 2021 12:43
Files that were supposed to be deleted: Files that had not been updated since 17:32 on December 3, 2021
[...]
Cause:
--
The backup script uses the find command to delete log files that are older than 10 days.
A variable name is passed to the delete process of the find command.
A new improved version of the script was applied on the system.
However, during deployment, there was a lack of consideration as the periodical script was not disabled.
The modified shell script was reloaded from the middle.
As a result, the find command containing undefined variables was executed and deleted the files.
[...]
Further measures:
--
In the future, the programs to be applied to the system will be fully verified and applied.
We will examine the extent of the impact and make improvements so that similar problems do not occur.
In addition, we will re-educate the engineers in charge of human error and risk prediction / prevention to prevent recurrence.
We will thoroughly implement the measures.That's not an argument against enabling it.
In bash, -o pipefail addresses this.
> and u is inconvenient when appending to PATHs
PATH should always be set. Try: env -i sh -c 'echo $PATH'
If you're prioritizing convenience over correctness, prepare to face the consequences.
> Every tool has its place, and dogma is often unhelpful.
Visual Basic's "ON ERROR RESUME NEXT" perhaps also had its place. That doesn't mean that using it is good advice.
If anything, I would consider the often cited wooledge etc. advice of not using -e/-u as dogma. Case in point: no one lost 77TB of data because they should not have used -e/-u.
Super not interested in a pedantic debate. It’s easy to armchair analyze. I found flaws in 55 codebases at Matasao, and yours is no exception.
e makes it super annoying to pass a variable number of args to a script, since shift will fail and cause an exit.
I do usually turn it on after, but you seem like the type to fail a code review if a script doesn’t start with it. I don’t think that’s a productive attitude.
> e makes it super annoying
I rest my case?
The original assertion was that under no circumstances should a bash script not begin with -e. I gave a circumstance (passing optional arguments), and said dogma is often counterproductive. I stand by all of those.
Let's agree to disagree and move on.
* I'm not sure what you mean by four PATHs, but if you really mean to be using unset variables for them, you should be using " ${V-}" or "${V:-}" syntax which does not fail. But again I don't know why you would do this other than maybe [[ "${1-}" ]]
* Variable arguments are still trivial with $#. Check (($#>3)), use while (($#>0)), etc
I also disagree that this is unproductive. With minor modifications/(adding :- or -), you can prevent a whole class of bugs (undefined variables). This woukd have prevent real-world issues such as in the post here as well as Steam when it wiped home directories since it ran (not sure the exact syntax) rm -rf $STEAMROOT/* with an unset variable
/bin/sh -eu
header in my scripts. It's a must-have.If you think your colleagues won't know this, "set -o errexit; set -o nounset" would be easier for them to search on.
(Via "3.4 Shell Parameters" → "3.4.1 Positional Parameters" → "4 Shell Builtin Commands", or searching the whole page for "-e".)
I'm intrigued to see that the report you link (which is in Japanese) mentions `find` and `bash` by those names, but doesn't contain the word `cron`. How does the report refer to the idea of a "cronjob"? Why is it different?
https://zenn.dev/mattn/articles/5af86b61004bdc https://iww.hateblo.jp/entry/20211229/file_lost_insident
適時: appropriate times, or as needed
Now I’ve known about this behaviour for a very long time and it always seemed very broken to me. It’s not how binaries work (at least not when I was doing that kind of thing).
So I guess bash or whatever does an mmap of the script it’s running, which is presumably why modifications to the script are visible immediately. But if a new file was installed eg using cp/tar/unzip, I’m surprised that this didn’t just unlink the old script and create a new one - which would create a new inode and therefore make the operation atomic, right? And this (I assume) is why a recompiled binary doesn’t have the same problem (because the old binary is first unlinked).
So, how could this (IMO) bad behaviour be fixed? Presumably mmap is used for efficiency, but isn’t it possible to mark a file as in use so it’s cant be modified? I’ve certainly seen on some old Unices that you can’t overwrite a running binary. Why can’t we do the same with shell scripts?
Honestly, while it’s great that HP is accepting responsibility, and we know that this happens, the behaviour seems both arbitrary and unnecessary to me. Is it fixable?
That is dependent on the OS. In this case wasn't the shell script just executed fresh from a cronjob?
I remember on Digital Unix - on an Alpha so this was a few years ago - that you could change a c program (a loop that printed something then slept, for example), recompile and it would change the running binary.
The description said that the script changed while it was running, so certain newly introduced environment variables didn’t have values and this triggered the issue.
My reading was that this was just a terrible coincidence - the cron job must have started just before the upgrade.
Regarding changing a C program, now you mention it I think that the behaviour you describe might also have happened on DG/UX, after an upgrade. IIRC it used to use ETXTBSY and after an upgrade it would just overwrite.
Not really behaviour that you want (or expect) tho.
As I understand it, the reason for ETXTBSY ("on some old Unices...you can't overwrite a running binary") was to prevent segfaults.
cp usually just opens the file O_WRONLY|O_TRUNC, which seems like the wrong default; Emacs for example does create a new file and rename it over the old one when you save, usually, allocating a new inode as you say. By default it makes an exception if there are other hardlinks to the file.
Btrfs and xfs have a "reflink" feature that allows you to efficiently make a copy-on-write snapshot of a file, which would be ideal for this sort of thing, since the shell or whatever won't see any changes to the original file, even if it's overwritten in place. Unfortunately I don't think you can make anonymous reflinks, so for the shell to reflink a shell script when it starts executing it would need write access to somewhere in the filesystem to put the reflink, and then it would need to know how to find that place, somehow. And of course that wouldn't help if you were running on ext4fs or, I imagine, Lustre, though apparently an implementation was proposed in 02019: https://wiki.lustre.org/Lreflink_High_Level_Design
Though the trade off is that all operation ceases on a full hard drive.
I don’t have a better solution, but it’s worth noting.
It doesn't cease all operation; other Emacs features work as they normally do. Bash, by contrast, stops being able to tab-complete filenames, at precisely the time when you most need to be able to rapidly manipulate your files. At least, that's the case with the default completion setup in a few recent versions of Ubuntu.
Oh noooo I just realized you could probably implement a shared library loadable module for bash `enable` that does the same thing... just fseek()s the fd...
*Runs for the hills screaming*
That's the route chosen by Windows for binary executables (exe/dll) and various other systems. Locking a file against writes, delete/rename or even read is just another flag in the windows equivalent of fopen [1]. This makes for software that's quite easy to reason about, but hard to update. The reason why you have to restart Windows to install Windows updates or even install some software is largely due to this locking mechanism: you can't update files that are open (and rename tricks don't work because locks apply to files, not inodes).
With about three decades of hindsight I'm not sure if it's a good tradeoff. It makes it easy to prevent the race conditions that are an endless source of security bugs on unix-like systems; but otoh most software doesn't use the mechanism because it's not in the lowest-common-denominator File APIs of most programming languages; and MS is paying for it with users refusing to install updates because they don't want to restart their PC.
1: Search for FILE_SHARE_DELETE in https://docs.microsoft.com/en-us/windows/win32/api/fileapi/n...
Naturally no one reads MSDN docs.
Also to note that other non-UNIX clones follow similar approach to file locking.
My experience is opposite, you can rename a locked file and place a new file with the name of the former one.
Depends on the lock type, I suppose.
this is incorrect, and is relatively easy to test:
$ strace -y -P /tmp/test.sh bash /tmp/test.sh
ioctl(3</tmp/test.sh>, TCGETS, 0x7ffc6daea580) = -1 ENOTTY (Inappropriate ioctl for device)
lseek(3</tmp/test.sh>, 0, SEEK_CUR) = 0
read(3</tmp/test.sh>, "#!/bin/sh\n", 80) = 10
lseek(3</tmp/test.sh>, 0, SEEK_SET) = 0
dup2(3</tmp/test.sh>, 255) = 255</tmp/test.sh>
close(3</tmp/test.sh>) = 0
fcntl(255</tmp/test.sh>, F_SETFD, FD_CLOEXEC) = 0
fcntl(255</tmp/test.sh>, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
newfstatat(255</tmp/test.sh>, "", {st_mode=S_IFREG|0644, st_size=10, ...}, AT_EMPTY_PATH) = 0
lseek(255</tmp/test.sh>, 0, SEEK_CUR) = 0
read(255</tmp/test.sh>, "#!/bin/sh\n", 10) = 10
read(255</tmp/test.sh>, "", 10) = 0
the reason why modifying a script during execution can have unpredictable results, not demonstrated in this test, is that Unix shells traditionally alternate between reading commands and executing them, instead of reading the entire file (potentially very large compared to 1970s RAM size) and executing commands from the in-memory copy. on modern systems, shell script sizes are usually negligible compared to system RAM. therefore, you can manually cause the entire file to be buffered by enclosing the script in a function or subshell: #!/bin/sh
main() {
# script goes here
}
mainBy reading in the whole file at once. Bash does not mmap shared the script it is parsing. You can see this behavior with
strace -e read,lseek bash << EOF
echo 1
echo 2
EOF
bash will read(), do its multi-step expansion-parsing thing and then lseek back so the next read starts on the next input it needs to handle. This is why the problems described in the story can happen.The other way to fix this is to simply use editors that will just make a new file and move over that file on the target on save. I believe vim or neovim does this by default, but things like, ed or vi do not. Emacs will do something similar on first save if you did not (setq backup-by-copying t) but any write after will still be done in-place. I tested this trivially without reviewing the emacs source simply doing the following and you can to with $EDITOR of choice:
!#/usr/bin/env bash
echo test
sleep 10
# evil command below, uncomment me and save
# echo test2
while running sleep, if changing the script causes things to happen, your editor may cause the problem described.about 15 years ago I experienced the same thing. An updater script based on rsync was trying to keep one nfs machine image in sync with another. However for what ever reason, the script accidentally tries to sync the entire nfs root directory with its own, deleting everything show by show in reverse alphabetical order.
At the time Lustre didn't really have any good monitoring tools for showing you who was doing what, so they had to wait till they hit a normal NFS server before they could figure out and stop what was deleting everything.
Needless to say, a lot of the backups may have been failing.
Options, as in I also found out about them the hard way.
Nice to see them completely own up to the mistake right away. I wonder who made the final call on doing so, companies admitting fault so transparently & immediately offering recourse seems pretty damn rare anymore.
Without the intent of sounding xenophobic, I wonder if it’s because it’s HP Japan where reputation is much more culturally important. US MBA’s admitting fault… haha…
* 真 makoto, a word often glossed as "sincere" but not identical in meaning: it's about the amount of effort you're willing to take on, not how "honestly" you feel something
Also, the culprit here is not HP proper but their consulting/SI wing HP Enterprise, which has a, uhh, less than stellar reputation for competence.
HP does consumer grade stuff only, while HPE does the enterprise side (not just consulting, in fact non trivial portion of HPE consulting arm was spun off and merged into DXC)
If I wanted to follow the trail of awesomeness what forest should I be sticking my nose to the ground in? :)
The trail that maintains the legacy of DEC and Silicon Graphics and Cray is in HPE (where I work). The Cray legend is still very much alive, but you can still detect the whiff of the the spirit that made HP and DEC minicomputers extraordinary.
[1] https://www.hp.com/us-en/printers/3d-printers/industries/hea...
[2] https://www.hp.com/us-en/industrial-digital-presses.html
Voluntarily stating 100% responsibility is consequential and not typical, smells politics.
I'm not a Japanophile, but find their culture fascinating.
[1]: https://theculturetrip.com/asia/japan/articles/sumimasen-beh...
Edit: This is not necessarily in support of the grandparent comment; rather, a caution against judging a culture based on historical anecdotes.
Also, going to war is inherently already a massive corruption of culture.
Sources? I have never heard of that yet.
Can I ask where your disbelief is sourced?
If you think that the domestic propaganda machine wasn't running at mach speed in order to shape public perceptions to what would be most beneficial to the Imperial Army and Navy, I don't know what to say. They definitely were, and they definitely pushed, from multiple angles, the idea that surrendering would result in a worse spiritual, material, physical, and psychological outcome than the alternatives (take your pick of whichever motivates you most effectively).
Not all failure results in capture, so one particular failure scenario does not speak of the general view on failure.
There are degrees of failure, some is considered to be beyond fixing.
And, wrt to samurais and failure, seppuku?
Yes, the parent comment is entirely wrong. The US culture is hyper accepting of failure compared to most other prominent cultures, including Japanese culture (which in fact does not tolerate failure very well at all).
The old white box PC shop I worked for, back in the 90s, quoted PCs to a Japanese-owned auto parts manufacturer. The Customer accepted the quote, paid an up-front deposit, and requested the PCs not be built and delivered until some construction at their site was completed in a few months.
In the meantime component pricing went down and speeds/feeds went up. When the PCs were built we ended up being forced to source higher clock speed CPUs and larger hard disk drives. It took some cajoling to get the Customer to take delivery. They felt they should pay more for the upgrades. (We were actually making more money on the deal even with the upgraded components anyway!)
My current company once pitched a support agreement to a Japanese-owned firm. We offered a discount for annual commitment versus month-to-month. I'd copy/pasted the month-to-month terms for the annual but forgot to alter the minimum notice period for ending the annual option. Both month-to-month and annual indicated a notice period of 30 days rather than the intended 180 days for the annual option.
The Customer's contact questioned the notice period being the same. He asked why they wouldn't opt for annual commitment, get the discounted rate, and also have the 30 day notice period. My partner, who had worked with Japanese firms in the past, responded: "We know you wouldn't choose the annual option if your intention was not to work with us for at least a year." The Customer agreed and we ended up getting the gig on an annual basis.
That's a really cute answer by your partner, I like it a lot, even ignoring that it manages to save face while paying serious respect to the purported customer.
What I found working with these teams, was that the desire to flip tables was quite strong after meeting with them. I tried to address their concerns, one point at a time. They were bewildered that windows could not mount (modern) lustre. Really bewildered. I offered to help rewrite their scripts in another (portable) language, so we could avoid these problems. Still they were bewildered.
They were not why I left. Merely a confirmation that my decision to leave was the right one.
And yes, your decision was the right one.
> * 真 makoto, a word often glossed as "sincere" but not identical in meaning: it's about the amount of effort you're willing to take on, not how "honestly" you feel something
While it's true that makoto can be translated as sincere(ly) in constructions like "makotoni moushiwakearimasen" (I sincerely apologize) (although this is written 誠に rather than 真に), it is unlikely that the word makoto would be used in a phrase like "a sincere apology" or in discussing how sincere an apology was, so I don't really think introducing the word "makoto" in your comment sheds any additional light on japanese culture surrounding apologies.
You could actually make the exact same comment about how "real" in English can also effectively mean sincere in "really sorry" and draw the same conclusions about American culture.
You could have an uphill battle against an unsympathetic judge in front of you if you sue anyway though.
The reason many US companies don't apologize is because they don't want to make restitution, and can get away without paying.
source? this is a popular theory among non-lawyers, but not, as far as I can tell, well-supported by the evidence. http://jaapl.org/content/early/2021/05/19/JAAPL.200107-20 has extensive citations including for their claim that "In theory, telling a patient about an error may make patients more likely to pursue litigation. In practice, however, bad outcomes alone are typically not reason enough for patients or their families to file malpractice claims."
——-
“hospital staff and doctors willing to discuss, apologize for and resolve adverse medical events through a “collaborative communication resolution program” experienced a significant decrease in the filing of legal claims, defense costs, liability costs and time required to close cases.”
https://www.natlawreview.com/article/you-had-me-i-m-sorry-im...
In my current company I have the opportunity to work with several large Japanese banks. Various members of our C-Suite and senior management travel occasionally for this purpose.
A sufficiently sincere apology with the appropriate level of ceremony can be the difference between maintaining and losing a substantial contract.
And this is why shell should not execute commands with "undefined" variables and give an error instead.
The style of apology is very nice. It is not extensive as some technical post mortem analysis that I've read, but all of the important things are here.
“A new improved version of the script was applied on the system.”
This is an incredible edge case. I'm amazed they hit this issue and just as amazed that they correctly identified that issue and reported on it.
This response is great, it's the exact opposite of the wishy-washy mealy-mouthed response to the lastpass security incident.
https://gist.github.com/robin-a-meade/58d60124b88b60816e8349... [^1]
And always, always, use ShellCheck (https://www.shellcheck.net/) to catch most pitfalls and common mistakes on this powerful but dangerous language that is shell scripting.
[^1]: I think this gist is better than the original article in which it is based, because the article also suggested changing the IFS variable, which is not that good of an advice, so sadly the original text becomes a bad recommendation!
Not really; the report doesn't mention any error in the script.
https://github.com/Mylab6/PiBluetoothMidSetup
While I could of done this in Bash.
1. I don't really like Bash
2. Python is much easier. I did challenge myself to only use Python's built in libraries, but aside from being unable to use Yaml everything works.
I can imagine in some environments you might not have access to a Python interrupter though...
Three things are certain:
Death, taxes, and lost data.
Guess which has occurred.
From https://www.gnu.org/fun/jokes/error-haiku.en.htmlWhat I like to do is have two directories I ping pong between when deploying, and a `cur` symlink that points to the current version. The symlink is atomically replaced (new symlink and rename it over) whenever the deploy process completes. Any software/scripts using that tree will be written to first chdir() in, which will resolve the symlink at that time, and thus won't be affected by the deploy (at least as long as you don't do it twice in a row; if that is a concern due to long running processes, you could use timestamped directories instead and a garbage collection process that cleans stuff up once it is certain there are no users left).
Just a note that "set -u" at the beginning of a bash script will cause it to throw an error for undefined variables. warning that of course this should be tested as it will also cause [[ $var ]] to fail.
If that's the case
[ -z "${VAR:-}" ] && echo "VAR is not set or is empty" || echo "VAR is set to $VAR"
will help test that condition
Is it maybe that they were editing or copying the file and a cron job kicked off?
printf "echo one\nsleep 3\n" > s1;(bash s1 &);sleep 1 && printf "echo two\n" >> s1
That's interesting. And changing the "sleep 1" to "sleep 4" make it only output "one". However, during deployment, there was a lack of consideration as the periodical script was not disabled.
The modified shell script was reloaded from the middle.
In my opinion, this is the wrong takeaway, and an important lesson was not learned.It's not an operator "lack of consideration".
The lesson should be "when dealing with important data, do not use outrageously bad programming languages that allow run-time code rewriting, and that continue to execute even in the presence of undefined variables".
If you use shell scripting, this is bound to happen, and will happen again.
"We'll use Python or anything else instead of shell" would fundamentally remove the possibility of this category of failure.
The shell scripts I write have fewer bugs than the Python code I see other teams churn out. But that's because I know what I'm doing. Don't hire people who don't know what they're doing.
Almost all languages allow run-time code rewriting. Some of them just make it easier than others, and some of them make it a very useful feature. If you're very careful, updating a bash script while you're running it can be useful, but most often it's a mistake; in Erlang, hot loading is usually intentional and often useful. Most other languages don't make it easy, so you'll probably only do it if it's useful.
Translated with www.DeepL.com/Translator (free version)