HNHacker News
TopNewBestAskShowJobs

Avernar

397 karma · joined November 5, 2014

submissionscomments
Avernar··on UTF-8 Everywhere (2012)
Which is why we have languages like Go where we can put those types of developers. Incidentally Go use UTF-8. Higher level languages like Go, Python, etc were designed so newbie and/or ignorant programers could do less damage.

When I was working on a project before Unicode we would switch our dev PCs to the other languages we supported. What a pain that was. Only issues we had was when a translated string was much longer than the screen space allocated to it. I belive Swedish was the main culprit. No problems with simplified and traditional Chinese as those were more compact. I have no sympathy for dev shops that can't get internationalization right. As with everything else in the corporate dev world management doesn't seem to want to hire/retain the more experienced programmers.

I think you have a gripe with my argument because you may be missing my point. If a high level language chooses to let a programmer index into a UTF-8 string at the byte level (for performance and other reasons) it's very easy for it to prevent the the programmer from slicing in the middle of a code unit.

The reason being is that the language function to slice a unicode string would either throw an exception or just advance to the next valid index. There wouldn't be a way for the programmer to slice a unicode string in the middle of a code unit.

Avernar··on UTF-8 Everywhere (2012)
> Plain text isn’t exactly a machine-friendly format.

Works fine for unix. :D

> Could you please name a Microsoft program that you think requires a BOM?

Visual C++ off the top of my head. It mangles UTF-8 string literals without the BOM in the source code.

> For me, Microsoft programs open text files just fine, with or without the BOM. But most *nix and osx programs show me garbage instead of BOM.

That's what I was trying to say about the BOM being prevalent on the Windows side of the fence. Some programs require it, some always generate it so most program now accept it.

On the unix/osx side everyone switched to UTF-8 so the BOM is redundant. Everything is UTF-8 so the silliness of this needs a BOM that doesn't need a BOM doesn't exist. Good example of what the "UTF-8 Everywhere" site is trying to promote.

Personally I really wish Microsoft would eventually fix their UTF-8 codepage. Would be so nice not having to convert to/from UTF-16 at the Win32 API boundary.

Avernar··on UTF-8 Everywhere (2012)
> I’m a Windows developer. In my world, a program should generate its output in whatever format user wants it to be.

When generating output for a user, letting them choose is a good idea. But for interop with other programs I leave it off unless the program needs it.

> Officially — definitely no, we both saw the spec on unicode.org.

The spec says the BOM is optional. Some Microsoft programs however require it.

Avernar··on UTF-8 Everywhere (2012)
There's a difference between what a program should accept as input and what it should generate as output. The standard just says to expect a BOM on input and suggests not generating on output. In other words "A UTF-8 BOM is a bad idea but some yutz out there stated doing it so we should ignore it on input". Someone else mentioned that the yutz was Microsoft.

I misread what you wrote about where you saw no idication that it was UTF-8. You were talking about places other than the web.

BOM for UTF-8 text files seems to be a Microsoft thing. Everyone else just defaults to UTF-8. But you can't be sure that it's a UTF-8 BOM or some other encoding. Most editors let the user overide what it is.

Why would you store text in a blob column? If a database can't handle UTF-8 in it's text column it needs to be fixed (or taken out back an shot).

Avernar··on UTF-8 Everywhere (2012)
Why should it crash? The proper procedure when validating a UTF-8 string is to replace errors with U+FFFD.

The term character has many meanings. Graphemes are characters and that's what most users expect, something that's displayed as a single graphical unit.

Avernar··on UTF-8 Everywhere (2012)
I consider changing a glyph to some other glyph(s) as corruption. Take an emoji flag character as an example. Split it between the code points and you end up with two boxed letters.

If you chop in the middel of a code unit then you end up with U-FFFDs. In both cases the visual representation has been altered.

As I wrote elsewhere it is easy for the slice routines of a language to check to see if the programmer tried to slice in the middle of a code unit and either return an error or just advance to the start of a code unit.

Avernar··on UTF-8 Everywhere (2012)
This drives me crazy. The Win32 API was designed for UCS-2. Then UTF-16 came out and the API was shoehorned to use it but as you said they still haven't caught all the places where it still thinks it's UCS-2.
Avernar··on UTF-8 Everywhere (2012)
Not at all. The language/library impementor should handle the details. My example was the argument checking of the slice function.

And indexing by code points doesn't solve the problem either. The majority of programmers don't know what a grapheme is or how to collate or sort unicode strings.

Avernar··on UTF-8 Everywhere (2012)
We're pretty much on the same page here. When you want to slice a string (because you can only display or store a certain amount) or you want to do text selection and other cursor operations you can't so it by code point. That's where you want to break at character boundries which are graphemes or grapheme clusters.

For parsing it's easier to just scan for a byte sequence in UTF-8 because you know what you're looking for ahead of time. If you're looking for a matching quote, brace, etc. you just need to scan for a single byte in your text stream. Adding a smart iterator to the process that moves to the start of each code unit is not necessary and will slow things way down.

I just gave JSON and XML as examples and not an exhaustive list. If you know the code points you are scanning for it's way more effecient to scan for their code units. The state machine in a paraer will be operating at the byte level anyways.

I have yet to see a good example where processing/iterating by code point is the better choice (other than the grapheme code of the unicode library).

Avernar··on UTF-8 Everywhere (2012)
Oops, made a mistake. The bit pattern is 10 in the upper two bits if you're not at the first byte of a code point and not 11.
Avernar··on UTF-8 Everywhere (2012)
From section 2.6 in the standard: "Use of a BOM is neither required nor recommended for UTF-8, but may be encountered in contexts where UTF-8 data is converted from other encoding forms that use a BOM or where the BOM is used as a UTF-8 signature."

Yes, it can be used to distinguish a UTF-8 stream but it's not recommended. One issue is you can't tell if the BOM is not valid text in some other non-unicode encoding.

I'm curious where you've encountered missing content-encoding headers or other OOB indicators where it wasn't because of programmer error or laziness.

Avernar··on UTF-8 Everywhere (2012)
Not impossible. If you slice in the middle of a grapheme then you get a corrupted string as well. You'll get an alternate glyph instead of a square box but it's still corrupted.
Avernar··on UTF-8 Everywhere (2012)
If by rune you mean code point that doesn't really gain you anything. If you are searching for something it's faster to just scan a byte at a time.

If you want to split the string into "characters" you need to do it a the grapheme level (multiple code points) and for that you need to use a unicode library. But that adds overhead when you're just scanning.

A JSON or XML scanner does not need the added overhead of advancing by code point or grapheme.

Avernar··on UTF-8 Everywhere (2012)
It's very easy to not get corrupted strings when byte indexing into UTF-8. In a while loop, if the index is not at the end of the string and the top two bits of the character at the index are both one, advance the index by one.

Or throw an invalid index exception if the top to bits are one if that makes more sense for the language you're using.

Avernar··on UTF-8 Everywhere (2012)
The unicode BOM is code point U+FEFF. The process of encoding it determines the order.

Encoded to UTF-8 it becomes EF BB BF. Encoding to UTF-16 big endian it will become FE FF. Encoding it to UTF-16 little endian it becomes FF FE.

Converting it back from UTF-8 always gives you U+FEFF since UTF-8 doesn't care about endianess. Converting it back from UTF-16 using the correct endianess gives you U+FEFF. Converting it using the wrong endianess gives you U+FFFE which is defined by unicode as a "non character" that should never appear in text.

Avernar··on UTF-8 Everywhere (2012)
No. Python 2.7 uses UCS-2 or UCS-4 depending on how it was compiled. Python 3 uses ASCII, UCS-2 or UCS-4 determined at runtime per string depending on the string's contents.
Avernar··on UTF-8 Everywhere (2012)
> You might want to slice the quotes off of a quoted string

That's why I really like dealing with UTF-8. As you said you can just index by byte instead of having to worry about code point boundries. This is because a code point will never match inside a larger code point.

So if I search for a one byte quote character it will never match the second, third or fourth bytes in a larger code point. Same with other control characters.

Works when searching for 2 or 3 byte code points as well.

Avernar··on This is the most demonically clever computer security attack I've seen in years
Depends on the type of VM. A VM emulating the same type of processor just runs the client code direct on the processor. What stops the client code from messing with the reat of the system, which includes the VM supervisor code, is the processor's protection circuitry. If client code does the exploit to kick the processor into supervisor or hypervisor mode it will be running client code in that elevated mode. Priveliged instructions would no longer trap to the VM's supervisor or host OS.

Now if the VM is emulating the client's individual instructions, which is the usual for running code for a different type of CPU, then it's a different story. Under pure emulation it wouldn't work. The exploit would have to know what the host CPU architecture was, get the emulator code to trip the extra gate. As you said, all that would do is put the VM supervisor code into the CPU's supervisor mode which it probably is already in if the VM is running bare metal without a host OS.

But if the VM use JIT conversion of the instructions for speed then the exploit becomes possible again.

Avernar··on Everything you know about interrupts is wrong
Minor nit pick: XT had one PIC, AT had two.
Avernar··on Python 2.7 Retirement Countdown
The bytes object is not a string type. It's treated more like a list of integers. It has very little string operations. Only in 3.5 did they add back the ability to % format.

Not to mention all the b prefixes it just becomes painful.

Avernar··on Python 2.7 Retirement Countdown
High quality unicode? Where every string with even a single emoji takes 4 times the memory. They should have used UTF-8 for their internal representation.

Putting that aside, my issue with 3.X is that I have to use the unicode type even when I'm manipulating byte strings.

Avernar··on The key differences between Python 2.7.x and Python 3.x with examples (2014)
"Now, in Python 3, we finally have Unicode (utf-8) strings"

This is not correct. Python strings are either ASCII, UCS-2 or UCS-4 depending on what code points they contain. Python 3 source code is by default UTF-8 and string literals are converted to the internal representation.

You have to encode strings into UTF-8 bytes to actually get UTF-8 when executing the program.

Avernar··on Wendy's plans to replace workers with self-service kiosks
> I can see how this could reduce error, but I think this has more to do with cutting costs than adding efficiency to the ordering process.

After using the touche screen at McDonalds only the first time was a little slower than going to the human. Now I can punch in my order super quick. As an added bonus I've seen the staff deliver the order to a person's table instead of being chained to a register.

Wish the Wendy's here in Canada would implement that soon. The human keeps try to rush through parts of the order. They're already asking about my drink when I haven't yet got the chance to tell them to hold the disgusting new pickels and red onions. Then it's the same conversiation about me not wanting to substitute fries for an extra item I ordered. Yes, I want them both.

Plus the fact that Wendy's here almost always has only one person on register unless it's exactly lunch or dinner time. And then they usually only have two registers anyways. During less busy times there usualy isn't anyone at the register and you have to wait for them to show up. So they've already cut costs on not having a dedicated register person.

Avernar··on A Ukrainian Hacker Who Became the FBI’s Best Weapon and Worst Nightmare
Ironic it works nice on desktop because it's glitchy on mobile. Had to reload several times before the wired banner would dock at the top of the screen. It would just float dead center of the screen the first few times.
Avernar··on Linksys WRT routers won’t block open source firmware despite FCC rules
The article title makes it sound like they're going against the FCC rules. The article itself however states they're complying the the FCC rules but are taking the more work for them option of just locking out the radio parameters portion.
Avernar··on Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan
Seems that a good rant was what you needed. :D

As I wrote in my other two posts tonight (more like morning, sigh) is that I tend to tune out emotional and aggressive writing styles. That's probably why my writing style tends to look aggressive. It's just the type of debates I tend to end up in (sigh, again).

So I apologize again if that got you upset at me.

Avernar··on Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan
> The AV software denied all disk I/O. If you have nowhere to put data, and no access to data, then you don't have a disk. You have a paperweight.

AV software does not deny all disk IO. It just denies write access to a file very briefly and then goes on to the next file as the article stated it was a scheduled scan.

So you do have a disk but a few files temporarily can't be written to (still can be read). The program will get an error code from the write function and can just try to write again.

> The disk I/O didn't fail. It was completely cut off by the AV program. There was no chance of it resuming until the scan completed, which could take far longer than the 5 minutes required to reboot the computer.

This is where you are incorrect. And AV scan does not lock the entire disk for the duration of the scan. It locks and releases each file as it scans them. Fire up process monitor from sysinternals and look for yourself.

Tried it with my AV scanner with a manual scan. Looks like mine doesn't even do a lock on most of the files when doing a manual scan. So at most the file just couldn't be deleted.

> My sole, singular point was this: Small programs are reliable programs. You can't have bugs in what you don't write.

I pretty much agree with you on that one. Smaller programs are more reliable than larger programs.

> What aggression do you feel I started with? I'm genuinely hoping to learn here.

I found it humorous that you saw aggression in my words and that wpietri saw aggression in your words. Me, I just learned to tune out that sort of thing in other peoples posts.

> I said that the spectacular crash led to pinpointing the AV scan as the source of the issue.

And that's what I was arguing about in my original post. You know what Root Cause Analysis is? If not read up about it here: https://en.wikipedia.org/wiki/Root_cause_analysis

My argument was that while the crash identified the AV Scan as a causal factor, it wasn't the root cause. From the wikipedia article: "Though removing a causal factor can benefit an outcome, it does not prevent its recurrence within certainty."

The root cause was that the programmer didn't handle the error code that his file was locked. There are many more causal factors that can trigger the exact same outcome: indexing service, backup program, shadow copy, etc.

Unrelated to our debate, the medical company only blamed the AV software and the IT Staff. Not one mention that their program had a bug.

The fact that their release notes warned against AV software means that they knew their program was deficient. That's what really pisses me off.

Avernar··on Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan
> Please consider dropping the adversarial attitude.

A debate is by definition adversarial. I do tend to be more passionate when debating certain topics. If I've come across to you as aggressive I apologize. It's just my style of writing and you can freely ignore any aggression you see in it.

> > Only if the programmer or his management were incompetent

> This is just short of a personal attack, which is against the rules.

That comment wasn't about you so it can't be a personal attack against you. It was about a fictitious programmer and his fictitious management used in our examples.

Attacking what you've written is not a personal attack against you. I will rip your words apart, try to prove they are wrong, show where you've either made a faulty assumption or an error in logic. That's what a debate is.

I will never, ever under any circumstances attack you. If you can see that distinction I will gladly continue to debate with you.

This whole thing basically ballooned from this statement of yours:

> The antivirus basically unplugged the disk. What can it do to recover? There's nothing to be done.

Those are the words I'm challenging. You have two points there. The first is that the antivirus unplugged the disk. While I know you're not being literal you're not being accurate either. It locked one or more files.

The second was that there was nothing the program could have done. To this I gave an example of a program that does handle this exact situation and more.

From your other comment replying to someone else but still quoting me:

> > Think of your torrent software. If you crank your firewall to block it while it's running it will not crash. If your disk fills up it won't crash.

> No one was saying it was okay for the program to crash.

This is the example program I'm talking about. I wasn't implying that you think it was okay for the program to crash. I was giving you an example of a program that can handle disk and network error conditions without the need to restart itself (automatically or manually) nor crash itself or the system.

> GPUs are bastards. They ignore what programmers want, almost by definition. And as someone who has spent way-too-many years wringing as much performance as possible from them, I assure you that this is a realistic characterization of a possible outcome.

I believe you in in that situation. GPUs have been designed with speed in mind and that makes for a very complex interface to them. But that reinforces a few arguments made by others in regards to this story, that being should medical equipment be using hardware not specifically designed for the purpose.

But going back to my comment of:

> > As for displaying random data, why would the programmer want to do this?

Based on your GPU comment above we may using different definitions of random data. I took your "random data" as "looks right on the screen but the numbers are wrong". If the programmer knows that his data source is temporarily unavailable, showing stale or corrupted data is the last thing he should do.

Avernar··on Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan
> You should never be able to confuse an error message for anything else.

Exactly. Which is why "Can't read file sensors.dat" is way better than just crashing. Crashing is one of the worst error messages you can get because you don't know what happened.

Avernar··on Medical Equipment Crashes During Heart Procedure Because of Antivirus Scan
Restarting quickly into a known state is not crashing. That is handling the error. How much of the program you restart is the question. Restart just a thread is better than restarting the whole program.

But their app crashed. And hard. It required a machine reboot to restart. While it returned the machine to a known state it wasn't quick.

And in medical software, all code paths need to be tested.

← PreviousPage 6 of 9Next →