HNHacker News
TopNewBestAskShowJobs

adamtj

762 karma · joined February 12, 2009

submissionscomments
adamtj··on Linux fork detection using thread specific keyrings
What if I use a library that forks or spawns threads? It would know nothing about these counters and wouldn't increment them, so I'd never know.

That's the neat thing about the original article: you can know if a fork happened outside of your own code.

adamtj··on Calculus for mathematicians (1997) [pdf]
You can find your epsilons and deltas in the definition of an open ball. They are just two different ways of writing the same idea. For example, in Definition 2.1, the 'h' in the open ball F is your epsilon and the 'h' in B is the delta. In an epsilon-delta proof, you show that for all |x - c| < delta, |f(x) - f(c)| < epsilon. The open ball B is just the set of all x satisfying |x - c| < delta and the open ball F is the set of all f(x) satisfying |f(x) - f(c)| < epsilon.

Any epsilon or delta that you choose implies a set of numbers satisfying those conditions. Those sets are open balls. By using them, you don't have to say things like "all 'x' such that ...". Which method you prefer probably depends on what you're more familiar with and how you tend to think. Open balls can be easier to visualize, if that's how you think.

Note that not just any ball will do. Closed balls are open balls that also include their boundary. That is, they use a less-than-or-equal-to instead of less-than. Which you use can make a big difference. An open ball on the real number line is just an open interval, an interval excluding its endpoints. It's easy to generalize: an open ball in a Cartesian plane is a circle excluding its border. In three dimensions, it's a sphere excluding its surface . . . and that's why it's called a ball.

adamtj··on How the Medium Editor Works
> 3. All visible edits should map onto an algebraically closed and complete set of visible content.

And suddenly I have a common-sense understanding of what the mathematical terms "closed" and "complete" mean. I can now use these ideas.

Non-closure is a problem that I've encountered often, but I've never had a name for it. If problems are obstacles, then names are the handles that let you grip them and move them out of the way.

adamtj··on Postgres Job Queues and Failure by MVCC
Doing anything after committing misses the point. If you can do that, you don't need postgres.

For example, suppose you mark an account closed, commit, and then enqueue an event to issue a refund from another system. It's possible that your process may crash or be killed at just the wrong time leaving you with a closed account but no refund.

So what if you enqueue the event before you commit? In that case, you might crash before committing which will automatically rollback. Now you've done a refund on a non-closed account.

Transactions make it trivial to guarantee that either both happen or neither do. There are other ways to get that guarantee, but they require more work and are more error prone.

adamtj··on Postgres Job Queues and Failure by MVCC
Unlogged tables aren't durable and might lose events even after a successful commit. If you're willing to make that trade off, then you don't need Postgres in the first place.
adamtj··on Vim Tips For Intermediate Users
This mapping makes my life much nicer:

  nnoremap Q @q
It turns 'Q' into a sort of enhanced '.' command that can repeat a sequence of multiple commands. To use it, first start recording a macro to the 'q' register. That's the easiest register because 'q' is also the command to start and stop recording a macro: to start recording you just simply press 'qq'. Then you make your changes and press 'q' to stop. Now you can repeat that sequence of commands with 'Q'.

It turns out that macros are really useful, but I would rarely use them because it's just so awkward to press '@' each time I want to apply one. Unfortunately, '.' doesn't repeat the '@' command to apply a macro. I would often find myself mechanically repeating things by hand, or else wasting time writing complicated sequences of :s/substitution/commands/. Now I use quick ad-hoc macros all the time.

The speed and convenience are the real key for me. You might call it synergy, or a tipping point, or the straw that breaks the camel's back. For me, 'qq' is just a little bit faster and easier than any other register. Repeating 'Q' is just a little bit nicer than typing '@q' followed by repeating '@@'. (It's also kinda nice that it's all the same key.) Macros are nice, but two carefully made choices that each are only slightly better than the alternatives combine to make something vastly more useful. In my mind, 'Q' is almost its own thing, separate from the idea of other macros.

Now that I use macros, I've discovered some useful patterns. One technique I often use is to include a final motion in my macro to move to the next place I want to change. For example, maybe I want to edit several lines in a row. I press 'qq', make my changes to the current line, then press 'j' to move down a line. Only then do I stop recording. After that, I can just hold down the shift key and press 'Q' repeatedly until I'm done. Even if my changes are simple enough that '.' would work, it saves me from making mistakes when bouncing back and forth: 'j.j.j.j.jj.'. Oops.

You can do that with any motion(s), and I often use 'n'. Sometimes finding some simple marker text on each problem line is easy, but using find-replace to make the actual changes requires a hideous regex. In those cases, I'll do the simple search, record a macro as I make the first change, and finally press 'n' to move to the next match. Then I just lean on 'Q' until I'm done.

Sometimes I'll also throw a 'zz' after the last motion to center the current line in the window so I don't have to look around to locate the cursor when a change scrolls the window.

Normally 'Q' enters Ex mode, but so far I haven't found much use for that. If that ever changes, I'll probably use a <leader> mapping to get Ex mode back. Using something even a little bit slower than 'Q' would destroy the effect. I forget where I originally learned this mapping.

adamtj··on SSH Agent Forwarding considered harmful
The last paragraph about eavesdropping is interesting. The agent doesn't actually give out your key, so an attacker can't use your key to decrypt any traffic. Instead, the attacker gets at the decrypted data as it passes between two ssh processes. Agent forwarding requires the data to be decrypted on the compromised proxy machine. That's the security hole.

The data is encrypted between the workstation and the proxy machine A, which we'll assume is compromised. The receiving ssh process running on machine A decrypts the data and gives it to a second ssh process, which re-encrypts it and sends it to the destination machine B. Because it's getting decrypted on machine A, anybody who controls that machine can view or modify the data.

With a proxy command, there is only one ssh process running on the compromised machine and it never sees the decrypted data. All it's doing is forwarding a TCP connection. The second ssh process (the one connected to B) runs on your workstation. The data is encrypted on your workstation and sent to B, via the TCP tunnel running on A. But the data passing through A is encrypted using ephemeral keys that A doesn't have. In this situation, controlling A is no more useful than controlling any of the other internet routers or switches that your data is passing through.

adamtj··on Null Pointer Dereferencing Causes Undefined Behavior
I'm sure he's right, but I'm not so sure he actually proved it. The real question is what happens when you deference a null pointer with '->'. He doesn't show us what the spec says on that, meaning there's a gap in his logic, leaving the quoted sentence in bold unsupported. I don't know but would bet that bolded sentence isn't even necessary. The undefinedness probably comes from the silence of the spec on what happens when you dereference a null pointer with '->'.

As he showed, a null pointer cannot point to any object, and since a lvalue that does not designate an object gives undefined behavior, saying '* podhd' would clearly give undefined behavior when 'podhd' is a null pointer.

However, instead of '* podhd' evaluating to a non-existant object, we have 'podhd->line6' asking for the object some particular offset from a non-existant object. It's entirely possible that a sufficiently insane spec might actually define such an operation.

What the spec says on that point is the key. That's where the undefinedness must come from, but he doesn't talk about it or quote from the spec on '->'. He just assumes it, which leaves his original point unproven, even if it is correct.

So why would anybody want to define legal behavior for dereferencing a null pointer with '->'? Well, some people start college funds for children who have yet to be conceived. If non-existant people can have account balances, why can't non-existant objects have properties? It's crazy, but it's not crazy like Intecal, where syntax errors are legal statements that you actually use for productive purposes. http://catb.org/~esr/intercal/ick.htm#Syntax-Error See also the statement "COME FROM".

adamtj··on The Man Who Broke the Music Business
Music consumers won't catch up to the decay that mp3's cause, _by definition_. Consumers only listen to music. The most sensitive, highly trained ears listening to the absolute best equipment in a perfect environment cannot reliably tell the difference between a decent mp3 and the original CD. Anybody who tries to distinguish guesses wrong as often as they guess right. Anybody who thinks they can tell the difference either got lucky the first time and quit testing, or is doing a flawed test (improper blinding, bad encoder, etc.)

The only way any consumer will ever notice that music is from an MP3 is if it's a bad encoding. That's the fault of whoever chose the encoding parameters, or picked an obsolete encoder. But even CDs aren't immune from stupid people. Lots of poor quality CDs are produced. I actually bought a major label CD with a song that clipped! It was mastered so loud that it clipped, and they didn't even bother to fix it. That's the sort of problem you can detect with software -- if you're deaf, because even my grandma could hear the flaw. How that got released, I'll never know. Yuck!

Only music producers may notice the "decay" you speak of. MP3 compression throws away data. It's data that no human ear can possibly hear, but when you're in the business of making music, you want to do things like process samples to change how they sound. It's possible to transform uncompressed audio data to bring out sounds that weren't audible before. Those inaudible sounds are likely to be thrown away by an mp3 encoder, making that sort of processing impossible.

But even CD quality PCM audio isn't lossless. It's perfect for listening to, but it's not great for processing. Music is often recorded and processed at higher bit-depths and frequencies before being compressed down to CD quality. You can't possibly hear the difference among studio quality 24 bit 192 khz, CD-quality 16 bit 44 khz, and a good mp3 encoding -- but the studio quality data gives you more processing headroom.

The photography world has analogous issues. Professional photographers and serious amateurs often shoot "raw". Their camperas are capable of producing normal jpeg files, but they don't use that feature. Instead they shoot and process massive, uncompressed files containing the raw sensor data. They then export the processed raw files to jpeg for publishing or printing. You can't see the difference between a raw file and a jpeg. (Actually, a jpeg straight off the camera often looks better than the raw file because the camera does a bit of processing.) The reason photographers bother with the big, unwieldy raw files is that they have more processing headroom. If a raw photo is too dark and there's a big black shadow in one corner, you can ofter brighten it up in postpressing and reveal the flowers that were there. If you do the same to a jpeg, which has much less information, the big black shadow will often brighten up into a big gray shadow. The "invisible" detail in the flowers has been thrown away during compression. The point is, the original unprocessed images looked the same. The differences are invisible in the same way that the differences between CD audio and good mp3s are inaudible. The differences only matter when you go to process the data.

adamtj··on The Man Who Broke the Music Business
Don't forget that the early encoders were pretty bad. It seems conceivable that a old encoders might have produced 320 kbit mp3s that were distinguishable from CDs in blind tests. Perhaps tempodox gave up on mp3s before they got good?
adamtj··on Visualizing matrix multiplication as a linear combination
Another way to look at it: A matrix is a linear transformation, and multiplying a vector by a matrix is how you apply the transformation. But linear transformations are really just changes of basis. How do you change your basis? You find the dot product of a vector with each new basis vector. And that's exactly what matrix multiplication is. When you multiply your column vector by a row in the matrix, you're finding the dot product, doing the projection in your change of basis.
adamtj··on Replacing Photoshop with NSString
This looks vaguely like xpm, but monochrome and vectorized. xpm is an image format that is also valid C code, so you can #include "image.xpm" in your C source code.

https://en.wikipedia.org/wiki/X_PixMap#Comparison_with_other...

adamtj··on Embracing SQL in Postgres
This ugly hack is not useful in general and only looks interesting here because the original example query is in bad taste. Simplifying it removes the tedium motivating the use of window functions.

For example, this is equivalent to the original query:

  SELECT sku, sum(price)
  FROM invoice_items
  WHERE date_part('month',created_at) = 9
  GROUP BY sku
See how the tedium disappears? In the original, the HAVING clause doesn't use any aggregate functions, which means it can instead be a WHERE. Removing the HAVING means we can further simplify the GROUP BY.

Of course, that only works because this example needs only a single month group. If you want multiple months and per-month price totals, then my rewritten query is no longer equivalent. It would need the more complicated GROUP BY. However, the window function version would also get more complicated in just the same way. The complication is inherent to the problem, not due to standards conformance.

Actually, what's really tedious is the verbose expression required to get the month and the fact that the author is repeating it. The standard provides "WITH" to avoid that duplication, and Postgres implements it.

Here's a reasonable version for multiple months:

  WITH items AS (
    SELECT *, date_part('month',created_at) AS month
    FROM invoice_items
  )
  SELECT sku, month, sum(price)
  FROM items
  WHERE month in (7,8,9)
  GROUP BY sku, month
But suppose we make the example even more complicated. Suppose you want sums over different time periods, like per-month and per-quarter. Then you can't use a GROUP BY and must use window functions and DISTINCT.

  WITH items AS (
    SELECT
        *
        ,date_part('year',created_at)    AS year
        ,date_part('month',created_at)   AS month
        ,date_part('quarter',created_at) AS quarter
    FROM invoice_items
  )
  SELECT DISTINCT
      sku
      ,year
      ,month
      ,quarter
      ,sum(price) OVER (PARTITION BY year)          AS yearly_total
      ,sum(price) OVER (PARTITION BY year, month)   AS monthly_total
      ,sum(price) OVER (PARTITION BY year, quarter) AS quarterly_total
      ,sum(price) OVER (PARTITION BY month)         AS month_number_total
      ,sum(price) OVER (PARTITION BY quarter)       AS quarter_number_total
  FROM items
  ORDER BY 1,2,3,4
adamtj··on Herculaneum scrolls unlocked using photon beams
How about this: "Scientists couldn't see what was written on the scrolls until they turned on some lights." The connotation behind "photon beams" more accurately conveys the accomplishment than the roughly synonymous "turning on some lights".
adamtj··on Signing in to websites with SSH
I love SSH, but I don't want to actually use SSH for this. That's not the important part. All I want is key-based authentication, and that doesn't need SSH. I want to be able to plug a security token into a USB port and be logged in, without even having to click a sign-in button. Reformat, reinstall, reboot, plug in security token, launch browser, type news.ycombinator.com and I'm already logged in.
adamtj··on Why the snow forecast for New York City was so bad
I disagree. When forecasters know with certainty that there will be 14 inches, that's what they should say. It's definitely more accurate. However, when they only know that it might go either way, they should say "2-16 inches", which is more useful and accurate in the face of real uncertainty. Even better would be "2-16 inches, probably 14".

A school administrator who sees a forecast of "14 inches" may simply cancel class the evening before, giving people time to prepare with certainty. That's nice, but not at the cost of a lost school day and calendar disruptions.

A forecast of "2-16 inches, probably 14" might cause that same administrator to wait and see. TV and radio would broadcast the news about possible school closings the evening before, so parents could prepare. They might have a bit more difficulty planning, but that's worth it if the school day can be saved.

adamtj··on OAuth1, OAuth2, OAuth..? (2013)
It sounds like maybe you don't understand the concepts.

Of course you're right that most people don't need to know exactly how encryption algorithms work. But, everybody needs to know what they do -- and what they don't do! That's a deeper level understanding than simply knowing if they're "secure" or not.

For example, too many people think that encryption gives you security. It does not. Encryption can provide confidentiality, but only if you also have integrity and authentication. Those three things are just the beginning of security.

One of the implications is that if you're using a self-signed certificate for HTTPS, you might as well not bother encrypting. If you don't reject a certificate lacking a verified signature, then you can't know that you aren't talking to a MITM instead of the server you think you're accessing. A MITM can trivially decrypt all your data, so why bother encrypting in the first place if you don't verify certs? Too many people ignore the certificates because they don't understand what encryption really gets them.

Many people also discount that danger because they don't understand how trivially easy MITM attacks can be. ARP spoofing is not hard. Some networking equipment is getting better at preventing it, but you can't always count on it. In short, it's best to assume that anybody else with a laptop in your local coffee shop can see _and modify_ all network packets you send. They don't necessarily have to break the wireless encryption to see them, either, so that won't keep you safe.

adamtj··on A positive leap second will be introduced at the end of June 2015
I disagree. Leap seconds are no more synthetic than timezones. As I'll explain, they are used for very similar purposes. Recognizing that means we have only one problem, not two. We can solve that one problem with existing code and processes with very few, if any changes. With no extra work, it also elegantly sidesteps the problem of being unable reliably predict when leap seconds will occur.

Leap seconds exist because we want the sun to be in the same position at the same time of day regardless of what year it is, despite the fact that the earth's rotation is slowing down. Timezones exist because we want the sun to be in the same position at the same time of day, regardless of where we are on the globe.

We can't reliably predict leap seconds in advance because the slowing of the earth's rotation is variable. We can't reliably predict timezone offsets in advance because legislatures are fickle things. We can no more command politicians to stop meddling than we can command the earth to stop slowing down.

Time is a natural phenomenon. It proceeds smoothly at a constant rate. (Ok, relativity. Still...) But, we want our clocks to measure more than just the passage of time. We want them to also indicate the position of the sun relative to where we're standing and what day it is. That causes discrepancies which give rise to time zones. Thus, we need to distinguish between a globaltime that is the same for everybody on earth and various localtime adjustments for convenience.

We generally use UTC as a globaltime standard. The problem with that is that UTC isn't smooth or constant because of leap seconds. Our localtime adjustments are difficult enough. We also have to deal with adjustments to our localtime adjustments. It's a hard problem, but a mostly solved one. Unfortunately, leap seconds mean we have the exact same difficulties in our underlying globaltime standard. We're solving the same hard problems twice, in different ways. I'm suggesting that instead we use a nicer globaltime standard and put all our adjustments and our adjustment adjustments into the existing localtime offset calculations.

We want timezones, so we need the complexities of implementing localtime. We might as well reuse that solution to deal with leap seconds too, since they're basically the same kind of thing.

You're right that with TAI there would be difficulties with calculating times in the future, but we already have those difficulties in a slightly different form. The nature of the difficulties would change, but the change would generally simplify things.

Currently, if you specify a UTC timestamp for a future event, the _duration_ between now and then will vary depending on how many leap seconds there are. However, the globaltime and localtime _timestamps_ would remain constant, regardless of leap secords. If instead we switched to TAI for our globaltime standard, both the duration and globaltime timestamps would remain constant, and only the localtime timestamp would change.

The unchanging timestamps might make it seem like UTC is better, but that's wrong. The timestamps only _appear_ constant. When those timestamps actually take place depends on leap seconds which can't be known very far in advance.

For example, suppose Alice and Bob are trying to coordinate an activity specified by a UTC timestamp. Neither can know in advance how long to wait, so they can't just set timers. If Alice accounts for all leap seconds but Bob misses one, his clock will be wrong and he'll start early.

If instead they used a TAI timestamp, then they wouldn't have any problems. They could just set a timer. Or, they could base their activities on their TAI globaltime clocks. Or, they could also use their localtime clocks. Bob missed the leap second, so his localtime is a second fast, but he would also think the event starts a second later. The errors would cancel and he wouldn't make a mistake. When Alice applied the leap second to her offset, she would also need to re-compute the now-changed localtime timestamp of the event, but computers are really good at such things. She could also write down the localtime timestamp of the event with an offset. If she applied the leap second to her localtime clock, but not her old pre-computed timestamp for the event, the clock and the timestamp would have different offsets. She would have to convert between the offsets in the same way she would have to convert between timestamps in different timezones.

Leap seconds are the same kind of problem as timezones. We already know how to deal with timezones. We should simplify things and use that one solution for both problems.

adamtj··on A positive leap second will be introduced at the end of June 2015
> Now, instead, imagine things being only a few seconds off, instead of entire hours.

No, and yes, and anyway it doesn't matter.

No, that wouldn't be the case. If you mess up the local->global conversion, you'd still be hours off, not just seconds. Realistically, we would probably need to define TAI2 = TAI + 35 seconds and then switch to that. Then the only immediate change would be that we'd stop adding the leap second flag to NTP updates. If you were doing UTC conversions correctly before, you'd automatically be doing TAI2 conversions correctly after, but if you mess up that conversion, you'll still be hours off.

But yes, such a change would introduce a new problem: Software or systems that are not updated to handle leap seconds in offsets might display localtime a second or two off.

Still, it wouldn't matter much. Non-TAI2 systems might display localtime a second or two off, but their clocks would still be correctly synchronized. Having an incorrect offset would be like inadvertently defining your own custom timezone. Nobody else would care about your timezone (whether it's a standard one or not), because you'd be communicating with them using global time based on your correctly synchronized clock.

adamtj··on A positive leap second will be introduced at the end of June 2015
Why don't we just switch from UTC to TAI and put the leap seconds in our localtime offsets? It makes all the problems go away, or else reduces them to problems we've already solved for handling daylight saving time.

If you think about it, leap seconds are no different than daylight saving time. The only differences are that daylight saving time usually has 1-hour granularity and is occasionally redefined by your government, where leap seconds have 1-second granularity and are occasionally redefined by Mother Nature.

We already know how to deal with repeating seconds and minutes (alternatively, with minutes or hours that are too long). Those happen during daylight saving time changes, but the ambiguity disappears when switching to UTC. We already record timestamps and communicate with UTC, applying a localtime offset for display or user input. That's a solved problem.

Further, we know how to change the localtime offset. Most timezones do that twice a year. We also know how to handle timezone definition changes because ignorant legislatures sometimes modify when the daylight saving time switches occur.

So, why don't we switch from UTC to TAI as an underlying time standard? When UTC leap seconds occur, we can treat them somewhat like a government redefining its timezones. US Eastern Time then changes from "TAI-05:00:35/TAI-04:00:35" to "TAI-05:00:36/TAI-04:00:36".

If two UTC clocks are synchronized, then one accounts for a leap second and the other misses it, the clocks will no longer be synchronized. If they were TAI clocks, they would still be synchronized and both would still record and communicate correct timestamps. It's just that one would display localtime incorrectly. But, you would immediately know that was the case when you see that the offset is a second behind. Missing a leap second would be like having your computer is set to the wrong timezone, and would be just as easy to fix.

adamtj··on GPGPU Accelerates PostgreSQL
This is very interesting, but not as a CPU saving optimization. I'm sure it does that very well, but that's not why it's Important. Rather, this seems to me like the next step toward the inevitable future of PostgreSQL as arbitrarily scalable, and as _the_ general query engine that ties together whatever physical data stores you happen to use.

It seems obvious to me that pushing the Foreign Data Wrapper layer with work like this is how we eventually break through the RDBMS scalability barrier of the individual host. In the future, I'm sure you'll see similar work where _the_ GPU won't be the GPU and the PCI bus won't be the PCI bus. Rather, they'll be _a_ host and the network. A database service (database cluster in Posgres's nomenclature) will eventually run not just on a single machine, but on a single cluster of machines. Instead of a cluster of machines for redundancy, you'll have a cluster of clusters.

Postgres is really two things in one: a physical layer of bytes in pages in files, and a logical layer of queries on tables of records. The most important piece in the future will be the logical layer. The FDW layer will naturally be extended and generalized until it is fully as powerful as the current physical layer. At that point, it can be made THE api through which the logical layer accesses data. The current physical layer will then be nothing more than the default implementation of that general API.

At that point, we can move whole or partial tables to other hosts. Perhaps the autovacuum daemon will gain a sibling in the autosharding daemon. The query optimizer will need to care not just about disk IO, but network IO and will need to start considering the non-uniform performance characteristics of different tables. Some tables will be driven by Postgres's default physical storage engine. Others will be driven by other RDBMSs, or by NoSQL key/value or document stores, or other data stores. They may be on the same machine or a different one.

Postgres will transform into a query engine on top of whatever data stores best fit your workload. I expect the query engine will learn about columnar stores and be able to mix those in a single query with the row stores, key/value stores and document stores that it already understands. PostgreSQL will be a central point through which you can aggregate, analyze, and manipulate any and all of your data. It needn't be intrusive or disruptive: you can still use a normal redis client for your redis store, but you can also use Postgres to manipulate that data with SQL and to combine it with other tables, whole RDBMSs, other NoSQL stores, spreadsheets, web services, or anything else. Maybe it will even make things like Map/Reduce frameworks redundant.

I don't typically follow Postgres's internal discussions, so maybe this is already being discussed and planned. Or, maybe it's so obvious that nobody even needs to talk about it. Or, perhaps I'm just some wide-eyed idealist who doesn't understand the fundamental problems preventing such a thing from ever being practical.

adamtj··on Why North Korea Fears 'The Interview'
First they came for the dumb comedy movies, and I did not speak out...
adamtj··on British Regulator Urges Home Births Over Hospitals for Uncomplicated Pregnancies
It's only uncomplicated until it's not. The solution to avoiding unnecessary surgery is not to physically locate farther away from all surgery.

Midwives can do their work in a hospital room down the hall from surgical facilities just as easily as they can in your bedroom 20 minutes away from major life-saving surgery.

adamtj··on Yik Yak Hack
What about ARP spoofing? Have all wireless routers ever built somehow gotten this right?

It's my understanding that unless the wireless access point is sufficiently smart, any device on the same network can claim to be the router and your phone won't know the difference. Even if an attacking device can't passively decrypt your radio transmissions, it won't matter because the router will actively decrypt and re-encrypt your packets before forwarding them on to the attacking device.

adamtj··on Std::string half of all allocations in the Chrome browser process
Reference counted copy-on-write strings are little landmines just waiting to blow your leg off should you venture into multi-threaded territory.

If you use copies of such a string in multiple different threads, you may find that simply creating a new copy of the original string or any of its subsequent copies can cause an incorrect reference count which will trigger a double-free and a segfault at some later time, possibly long after all non-main threads have ended.

You can't even lock all accesses to a string with a simple mutex. You have to also lock all descendant and ancestor copies, including all temporary copies, as when passing by value to a function. A COW string is basically a pointer to an internal shared data structure. You need to lock accesses to the shared structure, not the pointers to it, and you can only do that internally.

The only way to reliably use such a string with multiple threads is to make the internal reference count and buffer manipulations thread-safe. It may save you some allocations and copies, but the thread safety will slow things down. How much, I don't know. I wouldn't be surprised if thread-safe copy-on-write is slower than copy-always.

adamtj··on PostgreSQL’s New LATERAL Join Type
LATERAL allows a subquery or function to refer to the records that it's being joined to when computing results.

WITH RECURSIVE allows a query to refer its own results when computing its results. That may sound mind-bending, but it's really just poorly named way to do a "while" loop in SQL.

A WITH RECURSIVE has the form (base-case-query UNION ALL iterative-step-query). The query for the iterative step can refer to itself via the WITH alias. When it does so, it's actually operating on only those records produced by the previous step of the iteration. The iterative-step-query will execute possibly multiple times, stopping only when it doesn't produce any more records.

Here's the WITH RECURSIVE example from the Postgres docs, translated into Python:

  all_records   = []
  previous_step = [1] # base case, i.e. "VALUES (1)."
  while previous_step:
      all_records.extend(previous_step)
      # iterative step, i.e. "SELECT n+1 FROM t WHERE n < 100"
      current_step = [n+1 for n in previous_step if n < 100]
      previous_step = current_step
The confusing part is that the alias "t" in the SQL example means different things in different places. Outside the WITH RECURSIVE, "t" is equivalent to "all_records" in the example. Within the WITH RECURSIVE definition, "t" is the same as "previous_step".
adamtj··on Kill init by touching a bunch of files
I think cbsmith's point is that a a good language makes it so easy to correctly unwind the stack that it's almost harder to not do it right. If we could go back in time and do it over using exceptions and a better language, this likely would have been an exception from the start, which would have made this bug easier to fix. For example, Python's "assert" raises instead of exiting the process. The caller could just catch the exception from the failed assertion and deal with it, without need to update any other code. The problem is, we can't easily change languages now, nor can we safely retrofit exceptions. Hindsight is 20/20.

It's worth noting that that argument depends on the exception-raising assertion having been present from the beginning. The correctness of stack unwinding code can sometimes depend of which exceptions a function raises. A particular piece of code that is currently correct may break if one of the functions it calls is changed to raise a new type of exception.

adamtj··on When is a string not a string?
edit: Well, aren't I a fool. I just read the article more carefully and realized that a string isn't a string when it's not a string -- that is, when it's not well-formed. Of course weird things happen given illegal input. However, I'll leave this as most of it still applies.

A string is not a string when it's serialized and you can't distinguish between the abstract data and its physical representation.

UTF-16 can in fact represent all Unicode characters. To say otherwise is to demonstrate a very common misunderstanding -- a misunderstanding that I will free you from. "Unicode" really isn't hard, if you know the right way to think about it.

Unfortunately, it seems that Microsoft doesn't know the right way to think about it. .NET "strings" aren't really strings at all. They're arrays of short integers, not characters. The internal physical representation leaks through the abstraction. The author of this article stumbled upon one place where the abstraction leaks and is surprised because, like Microsoft, he doesn't understand the distinction between strings and their serialized representations.

What this article describes is essentially this: take a Unicode string and serialize it to bytes as UTF-16, then deserialize those bytes with UCS-2. UCS-2 is a different encoding than UTF-16 and mixing them can easily mangle the original string. If you then serialize both the original string and the mangled string with UTF-8, you'll get different bytes for each.

Serializing a string with UTF-16 then deserializing with UCS-2 is like trying to save an image as a .JPG then load it as a .BMP. If it happens to work, you won't have the same image you started with. It will be mangled. This should surprise no one.

The only sane way to think about Unicode is this: A string is an abstract data structure. It is like a object of some class: you can't write it directly to a file or send it across a network without first serializing it to some physical representation as a sequence of bytes. There are many possible ways to serialize a particular object to a sequence of bytes and there are many ways to serialize a particular string to a sequence of bytes. A byte is not a character any more than a byte is a Date object.

As you might expect, you must use a matched serializer/deserializer pair or the object you load may not be the same as the object you saved, if it works at all. That's what happened in this article: he saved a string in UTF-16 format, then loaded those bytes with a UCS-2 reader. Like JPG vs BMP, the results were scrambled.

A "string" is an abstract sequence of characters.

A "character" is a named and numbered element of a particular "character set". Some characters are present in multiple different character sets and are often assigned different numbers in each.

A "character encoding" is a bi-directional mapping from a physical sequence of bytes to a abstract sequence of characters, i.e. a string object. A character encoding is typically tied to a particular character set.

"Unicode" is a widely used standard that defines a particular character set and a number of different character encodings such as UTF-8 and UTF-16. The numbers assigned to each Unicode character are called "code points".

If you can understand the distinction between the abstract data that a string represents and its many possible physical representations, you can peel back the layers of confusion and make sense of any character encoding problem.

Conclusion: Text is not hard, if you have the right mental model for it.

adamtj··on Let's Encrypt: How It Works
How is this not really bad?

Suppose I somehow get control of a machine on the same network as a server that doesn't use Let's Encrypt. I can probably ARP spoof it pretty easily. If it has a signed certificate from a different CA, I can't MITM it without people noticing. Or I couldn't, until now! Because I can now "control" the server's responses, I can easily get Let's Encrypt to authorize a new key pair giving my control over that domain. I can now generate valid certs for my MITM keys.

If the server were already using Let's Encrypt, I couldn't create a new controlling key pair. The "How It Works" doesn't talk about that, but search the RFC for "recovery token". But if the server isn't using Let's Encrypt, what would stop me from doing this?

Of course, I'm actually on the other end of it. To project myself, do I really have to at least register a controlling key pair with Let's Encrypt and any other browser-supported CAs that adopt this protocol?

adamtj··on Kill init by touching a bunch of files
Exceptions are not a silver bullet. One can't simply turn an assert into an exception and expect things to work. That's a great way to add new and tricky bugs unrelated to the original issue.

If an exception crosses a stack frame that isn't expecting one, things could be left in an inconsistent state. Invariants may vary, constraints may be unconstrained.

To retrofit exceptions into the existing code at this point would be very difficult, as you would have to audit every caller of the function in question, a nd the callersof those, and on and on...

Imagine problems like Apple's "goto fail" SSL bug, but more obscure and harder to find.

To have used exceptions from the beginning would require an amount of work roughly equivalent to returning error codes. They aren't fundamentally different from error codes in that way. Well, unless you're a cowboy who ignores errors.

Exceptions wouldn't have solved anything here.

← PreviousPage 2 of 9Next →