lambda

What's wrong with unbounded? The amount of text you're rendering is unbounded (except by resource constraints). Combining diacritics are just another kind of text, that happen to lay out vertically rather than horizontally even in horizontal layout.

What do you mean by "getting the number of characters actually displayed on the screen"? Do you mean number of glyphs? Number of grapheme clusters? The number of glyphs is dependent on your text rendering system and fonts; even with plain ASCII, you may have multiple codepoints rendered as a single glyph, such as in ligatures like "ffi" in certain fonts. The number of grapheme clusters can be computed by the given algorithm along with some tables of character properties; it's not particularly simple, but trying to represent languages that were originally hand-written on a computer is not simple: http://unicode.org/reports/tr29/#Grapheme_Cluster_Boundaries

I think that moving the world away from HTML and XML as markup languages is likely to be considerably harder than simply defaulting to using UTF-8 in any new code for which there isn't already a natural encoding to choose based on the platform or APIs you're developing on.

Furthermore, even beyond the markup, the vast majority of written content is in the Roman alphabet, of which most characters are in the ASCII range, and those that aren't fit into 2 bytes of UTF-8. 55% of text on the web is in English, then languages using the Roman script or other scripts in the 2-byte UTF-8 range make up 21 of the next 25 most commonly used languages on the internet; of the top 25 languages, only Chinese, Japanese, Korean, and Thai are in ranges that require 3 bytes in UTF-8 for the majority of their characters (https://en.wikipedia.org/wiki/Languages_used_on_the_Internet).

So for the vast majority of text that is processed, UTF-8 is dramatically more efficient than a hypothetical UTF-24 or the real UTF-32. Now, you might say "well, it should all compress away anyhow", but when dealing with text in RAM, it is generally not compressed (and that would make handling it far more complex than just using UTF-8), and memory capacity, latency, and bandwidth can all be important.

Trying to deal with text as fixed-width characters is simply incorrect. Almost every non-trivial means of handling text needs to support arbitrary length strings, needs to deal with clusters of more than one codepoint as a single unit, and needs to iterate over the characters linearly at least once (and can then store byte offsets for random access later on). Dramatically increasing storage requirements of text for the non-benefit of being able to deal with fixed-width codepoints just doesn't make sense.

This discussion is about what encoding is best to choose for the low-level string types in languages and libraries, as that's what the original article was about.

If your language already uses a Unicode-compatible string type, you should probably use that.

If you are designing a new language, rewriting a language's basic string primitives, or writing an API or application in a language that doesn't have a well-standardized Unicode string type like C or C++, then it is probably better to pick UTF-8; if your app is necessarily single platform and tied to a platform API, choose that API's encoding, but for anything cross platform, the encoding is going to differ between the platforms so choosing UTF-8 and then transcoding at the API boundaries on the different platforms is fine.

By the way, most languages do have the concept of a byte array, and you can provide UTF-8 string operations on top of that; but as I said, in general, you should be using the language's native string type unless you have a very good reason not to.

The original article that we're discussing does mention one of those very good reasons; in Python, it uses an 8-bit Latin-1 encoding if the characters can fit, upgrades to 16-bit UCS-2 if they can fit in that range, and finally upgrades to 32-bit UTF-32 if they don't fit in the UCS-2 range. But that means that if you're appending strings, and one of them contains an emoji (for instance, in a web templating system that encounters an emoji in a comment), that can mean that all of a sudden your string has to grow to four times its original size, which can cause serious problems; so one workaround for that may be to actually write everything out to a UTF-8 encoded string (which in Python can be represented as a byte array) rather than trying to append strings together and hitting that behavior.

Furthermore, several languages already do use UTF-8 as their internal string type; Rust, Go, and Ruby all use UTF-8 as their default or only string type.

And by the way, Rust and Go both use slices to handle the use case you describe; a slice is represented as a pointer to the first byte of interest, and a number representing how many bytes it extends. I don't know about in Go, but in Rust the API guarantees that all string slices created from a string begin and end on proper codepoint boundaries. Go: https://blog.golang.org/slices, Rust: https://doc.rust-lang.org/std/primitive.str.html.

But why bother with a fixed width 128 bit encoding? We have perfectly fine variable-width encodings, that take up 8 bits per character for the most common set of characters encoded, the ASCII range (even for CJK text, the fact that most markup languages like HTML and XML use ASCII means that actually, the bulk of documents fall within the ASCII range), and can handle arbitrarily complex stacks of diacritics.

How would it be helpful to waste a huge amount of space for the vast majority of text, impose additional limitations on the number of combining characters, just for the benefit of fixed-width encoding of graphemes, which even in itself is not all that helpful since for most applications, you are either going to need to scan through the text linearly, or you could use byte offsets in the variable width encoding just as well as you could use grapheme offsets in a fixed width encoding?

Note that my question included "...that is actually correct" as a condition.

Truncating between code points to insert an ellipsis is not correct. Not all characters have the same width, even in a fixed-width font; for instance, combining characters have zero width. Truncating between a base character and a following combining character makes no sense. If you're truncating the word "exposé" to fit into six characters, it makes no sense to count that as 7 characters, truncate the accent, and wind up with "expose...". (Yes, normally that will be represented as a pre-composed character, but there are base-character/diacritic combinations that have no pre-composed character, so you will necessarily get a combining diacritic).

Truncating based on character count can also mean that you exceed your width requirement; CJK characters are twice the width of most alphabetic characters.

Additionally, most text is displayed using variable width fonts, where truncation should depend on the precise sum of the widths of characters, but even text that isn't can have the previous two issues. These mean that to do truncation properly, you need to iterate over the characters counting width until you exceed your required width in order to truncate, then truncate at the previous grapheme boundary, which requires walking through the string in order and can be done just as efficiently in a variable width encoding as in fixed.

We have not yet developed writing systems for all of the world's languages. So there is no practical bound you can put, because it is possible that in a future development of a writing system, there will either be entirely new characters invented, or new combinations of existing characters and diacritics.

In addition, since the existing Unicode standard does not impose a bound, and implementations support stacking of large numbers of diacritics, people have used combinations of extremely large numbers of diacritics as a stylistic device, for example: http://stackoverflow.com/a/1732454/69755. Since examples like this exist in the wild, they need to be supported properly, and so the fundamental string types and operations in string handling libraries should be able to handle it.

Why would you want constant time access to code points? What can you do with constant time access to code points that is actually correct?

Every time someone tries to promote UTF-16, UTF-32, or some imagined encoding like UTF-24, they bring up constant time access to code points, but I have never heard of a reason why you would want that.

Pretty much every use case I have ever heard for constant-time access can be handled just as well by constant-time access at the code unit (byte in UTF-8, 16 bit value in UTF-16, 32 bit value in UTF-32) level. Matching text exactly? Matching works just as well at the code unit level. Doing any kind of fuzzy or regular expression match? You're going to need to iterate over each item anyhow to normalize it or classify it. Need to store offsets? That works just fine at the code unit level.

Most of the other suggestions I've heard for what to do with constant-time access to code points is incorrect when you consider combining characters, normalization, different character widths, need for linguistically appropriate word splitting, etc. If you're doing something that doesn't take these into account, why are you using Unicode instead of just ASCII?

In addition, for the ASCII range, UTF-24 would take up 3 times the space as UTF-8, and the vast majority of text processed is actually in the ASCII range due to verbose markup formats like HTML, XML, etc. Plus if you do anything with the code points in UTF-24, you need to do a bunch of bit-fiddling to move them into 32 bit alignment, so as far as actually decoding the individual code points, it's pretty much a wash with UTF-8.

I collected up a number of nits like this, plus some more meaningful transcription errors, into a branch, and have sent a pull request: https://github.com/tdop/tdop.github.io/pull/2

Can you link to the scanned PDF and your GitHub repo? I've noticed a few places with mismatched brackets, such as "(....]", and wanted to check to see if they were mismatched in the original or if that's just a transcription error.

edit: Found the original at http://hall.org.ua/halls/wizzard/pdf/vaughan.pratt.tdop.pdf and your repo at https://github.com/tdop/tdop.github.io. Am reading the paper now, it's not one that I'd read before so thanks for posting it, and will send a pull request with suggested fixes when I'm done.

Er, yeah, of course you're right.

Adding that factor in will mean adding in even more fake, hand-wavy numbers, but doesn't really change the end result; whatever value you end up with, if you double the weight of the batteries, you will need to displace twice as much air (or move it twice as fast) to compensate, which means you'll need to use up twice as much energy, leaving you with the same flight time in the end.

Yes, another approach is to not have to carry all of the energy with you. Solar charging is another possibility.

Price and safety, I would bet.

Let's work out how that would work, in a simplified model of the world.

Let's say you have a 1000mAh battery at 10V that weighs 100g (these numbers have all been made round for purposes of easier arithmetic). Everything else on your drone is weightless, and your motors have perfect efficiency. To keep this aloft, you need to overcome 9.8 m/s^2 that is applied by Earth's gravity; but since we're simplifying things, let's just round that to 10 m/s^2. 10 m/s^2 * 100g is 1 N, which means it requires 1 W of power to keep that battery aloft (resist the force of gravity). 1Ah * 10 V = 10 Wh, so that means you could stay aloft for 10 h (remember, these numbers are unrealistic, we have ignored any kind of efficiency, or the weight of the rest of the drone, etc).

So, lets use two batteries. Now we have 20 Wh of energy available, but we're holding up 200g of battery, which requires 2 W of power to stay aloft, so again, we can stay aloft for just 10 h.

And remember, all of this was ignoring any notion of efficiency or the weight of any other part of the craft to support this. As you increase the weight, motors will need to get more powerful to be able to actually keep it aloft, which means they need to get bigger, which is extra weight that you need to support that isn't giving you any extra energy. So the real world is actually even worse than our idealized model; as you add on more battery, only a fraction of the increase in weight adds more energy, so you will actually see shorter flight times.

Now, of course, there are designs for which increasing the amount of battery can lead to increased flight times. If the power supplied by the motors is sufficient to sustain the extra weight, so they don't need to be scaled up, then increasing the amount of battery relative to the rest of the weight of the craft can lead to higher efficiency as there is less relative "dead weight" (weight besides the batteries, that is not carrying energy). But that can only take you so far, which is why the flight times of drones plateau at a certain point and you can't fix it by just adding more battery.

This is why you don't see electric helicopters carrying people. The energy to weight ratio for batteries is too low; only fossil fuels have a sufficient energy density.

Nope. We're pretty much limited by battery technology on that front; you need bigger batteries to provide more power, but those bigger batteries mean more weight, which means more power needed just to stay aloft. For a battery powered multicopter, I think an hour or so is about the highest you can get (I don't think I've seen any that claim over 40 minutes), and as you point out, most drones are closer to the 15 to 20 minute range.

Of course, there are other designs like fixed wing planes which can be a bit more efficient though more complicated to navigate in confined spaces and without the ability to hover, and you can use internal combustion engines which can get you much longer flight times due to the higher energy density of their power source. But I don't think you're going to find any of the affordable, battery powered multicopters that get you anything over an hour (and even then, I would set my expectations at more like a half-hour for anything affordable in realistic conditions).

  Leaving this message in place as an example of how  
  thinking about time and calendar-related programming is 
  easy to mess up.
		

Indeed, time and calendar related programming is quite tricky, even specifying what the behavior should be can be tricky and very non-intuitive.

A lot of people who believe in FC and it's variants believe that there's a big conspiracy to silence the voices of their children. They don't think it's relevant that there was a facilitator who was just "supporting his arm" or "providing contact" or something of the sort. So they are unlikely to do this; they want to promote a notion that the child is producing the text, not the facilitator.

Note that this was most likely written by his facilitator, not by him. It was written via a technique called "Rapid Prompting Method", which appears to be a variant of facilitated communication, a method that has been debunked as simply being the facilitator guiding the child, either consciously or subconsciously, to type things that the facilitator is thinking, not that the child is thinking.

See https://en.wikipedia.org/wiki/Facilitated_communication for more information. I also recommend the PBS Frontline documentary "Prisoners of Silence" (transcript: http://www.pbs.org/wgbh/pages/frontline/programs/transcripts..., video: https://www.youtube.com/watch?v=HXw8Ksvyt5Y), which does a good job both debunking it, and demonstrating why facilitated communication can be quite harmful; there was a rash of incidents in which facilitators imagined sexual abuse allegations from children, leading to substantial repercussions for innocent parents.

That doesn't sound like a terribly hostile tone. Yes, there was a little bit of hostility, but nowhere near the level that would lead a support person to give up on helping you.

Remember, people in support are used to a somewhat hostile tone. People get in touch when something is wrong, and so it's fairly common for people to be frustrated. If they just responded with "sorry, can't help you" to every support query with a slightly hostile tone, they wouldn't be providing much support.

And yes, it is also good to try and assume good faith and not vent your frustration at a support person who had nothing to do with whatever issue it is that you're frustrated about. But it's a fact of life in support that you'll have to deal with some unhappy people, so you need to be able to deal with it.

> The other alternative is you download source, at which point the Makefile or any other piece of the build that you execute can do whatever the shell script would have done

Part of the problem with this is that since it's a bootstrapped compiler, and the only one for the language so far, "downloading source" mean you need a binary to compile it with, which devolves to the same problem.

Yep, the dropping of the sigils was a big help in my actually giving the language a chance. It's been on my radar for a while, but went through a pretty ugly period for a while before coming out much better than before.

There was also just a sense of "OK, I get what these things do, but when should I us an @ vs an ~ vs a & in my API", and now I feel like it's much more clear what you use the different reference types for.

It is possible, with a little manual hacking of the linker and some trickery with storing strings in ELF headers, to get Rust to produces 151-byte executable.

Yes, a simple "Hello World" winds up statically linking a fairly sizeable standard library, in particular anything dealing with strings has to involve a certain amount of UTF-8 manipulation and the formatting and I/O have a lot of features and thus likely fairly large code size. But for the most part that's just fixed overhead, and you can opt out of using the standard library if you want and produce smaller executables.

You're right, that site should be updated with those as other parts of the ecosystem that aren't yet mature.

There's work on non-blocking IO at https://github.com/carllerche/mio, which is looking like one of the more promising approaches, as well as https://github.com/carllerche/eventual which provides higher-level abstractions over it.

There has been some discussion of adding some language support for something like async/await/yield from other languages, but nothing concrete in the near future on the language front.

So, there's usable third-party libraries for it (at least, if you're on a Unix like platform), language support is in the realm of "it would be nice someday".

Docs are maintained here: https://github.com/rust-lang/rust/tree/master/src/doc

The book is in the "trpl" directory: https://github.com/rust-lang/rust/tree/master/src/doc/trpl

The standard library docs are maintained in doc comments in the appropriate places in the source: https://github.com/rust-lang/rust/tree/master/src/libstd

You can clone the repository, modify it locally, and build and test the docs (yes, the docs get tested, most of the code samples are actually executable tests), or you can just edit them online in GitHub's editor and send a pull request that way if you're just making documentation suggestions that won't need any tests.

The reason that you see so many multicopters for the smaller ones is that they are much mechanically simpler, and thus cheaper to manufacture. All you need mechanically for a multicopter is a number of independent rotors that you can vary the speed on.

A traditional helicopter has a very complicated assembly to allow powering a rotor that can tilt forward, backwards, and to the sides, as well as change the angle of the blades. This is a lot more expensive to produce.

It is more efficient, in a power to weight ratio sense, to have only a single motor, which is why you see this design on full-sized helicopters. But for smaller, cheaper drones, you don't need as good of a power to weight ratio to make it worthwhile, and the cheaper simpler design makes them easier to mass-produce and sell to amateurs or professionals on a budget.

The things that probably have led to an explosion of them recently include electronics getting cheaper, and control systems getting better. Regular helicopters can be piloted by a skilled operator with some training. A mutlicopter needs a automated control system to keep it balanced. As electronics for the control systems have gotten cheaper, as well as MEMS sensors (accelerometers and gyroscopes built into ICs), these control systems have become cheap and easy to integrate into mass produced products.

In addition, battery technology has been improving, as has electric motor technology. While previously only the single big fossil fuel engine was the only viable power source, as battery capacity has increased and electric motors have gotten more efficient, it's more viable to have small electric copters.

Another thing that makes these more accessible to casual amateurs, and again makes mass production more viable, is that most people now carry around a flexible input device with wireless control capabilities. That means that you can sell devices without a dedicated controller, can do things like streaming video back from the drone, and so on, increasing the range of things that you can do with one.

Other technological improvements also help feed into this. Having widely available, small, light HD and 4K cameras means that you can actually do something useful with these drones; back when to do any real filming you needed a big, bulky camera, they weren't as efficient for getting aerial shots, but once there are small, light, high quality cameras, it becomes feasible.

Ah, while I was writing that sctb replied explaining that they had changed the title.

The title currently reads "Ripple fined for acting as a money services business without registration", and the complaint says "Ripple Labs willfully violated several requirements of the Bank Secrecy Act (BSA) by acting as a money services business (MSB)and selling its virtual currency, known as XRP, without registering with FinCEN".

That looks like the title matches the contents. Did the title change since you wrote the question? It is important on Hacker News, when addressing anything about the title of a submission (or about the precise URL of the submission), to quote the title or URL you are talking about, as moderators sometimes silently change the title or update the URL to point to the actual underlying story rather than blogspam.

It's a concept that's been done before: http://i.imgur.com/wteBl.jpg

When you are on the job for the government, you are acting as a government representative and not a private person. You can be fired for what you say while on the job.

However, the police unions would never allow such a thing.

Ah, thanks for digging that up. I was not aware that they didn't support fractions in exponents; I don't think I had ever tried them for that. Luckily they do give warnings, though only if you have already set the precision (via "bc -l" or something like "10 k" in dc).

Sadly, since dc doesn't have a math library, it looks like you can't use the "e" function workaround unless you implement it yourself.

So, what is actually new here?

They had already announced the various parts of this, and the only link I see in here is a GitHub repo with links to other GitHub repos that we've already seen. Is there anything new in this announcement?