HNHacker News
TopNewBestAskShowJobs

andyjpb

1,389 karma · joined April 23, 2010

submissionscomments
andyjpb··on Why do we fall into the rewrite trap?
The Netscape business model was largely about selling their FastTrack Sever (for "eCommerce") and per-seat browser licensing to for-profits.

They had a reputation for giving away betas and pre-release copies on their FTP servers.

andyjpb··on Software is a focus intensive industry
This argument is built out of rough numbers that came to hand using quick searches. It's not particularly accurate but gives an idea of the orders of magnitude involved.

Google has approximately 100,000 employees. Not nearly all of them are software engineers but, for the sake of this argument and to be generous, let's assume that they are.

Let's also be generous in assuming that only people in the US, and nowhere else, use it. That's about 330,000,000 users.

OTOH, the US army has approximately 1,300,000 service members and 800,000 reserves as well as approximately 330,000 civilians.

So (ignoring physical suitability) the potential labour power of the US army is approximately an order of magnitude greater than Google. Moreover, "labour intensive" usually means "people who go out to do physical things": the work scales with the number of hands you can bring to bear. Combat aside, military forces are often involved in work that requires lots of people to do things in "meatspace":

* Rescue operations * Medical assistance in impoverished areas * Food & humanitarian relief * Security at embassies and other locations * Policing in volatile areas * Natural disaster relief * Law enforcement * Piracy and drug interdiction

When the software engineers aren't busy thinking about software, they carry cups of tea between the kitchen and their desks (well I suppose they're probably thinking about programming then as well ;-)).

Finally, Google currently has a big operation, but like many startups, a pretty big product (the original search engine) was built by ~20 people. With those people they crawled almost every site on the web. If you wanted to go and read every document in every filing cabinet and index that then you'd need a labour intensive force bigger than the US army.

Here are the references I used:

https://www.statista.com/statistics/273744/number-of-full-ti...

https://en.wikipedia.org/wiki/United_States_Armed_Forces#Per...

https://www.goarmy.com/careers-and-jobs/army-civilian-career...

https://www.military.com/join-armed-forces/military-missions...

andyjpb··on I spent 22 years in prison for a crime I didn’t commit
As I understand it, it was due to some problems that were repeatedly encountered when trying to prosecute members of the Provisional IRA.

http://news.bbc.co.uk/onthisday/hi/dates/stories/october/20/...

andyjpb··on I spent 22 years in prison for a crime I didn’t commit
The right to silence was changed to this in the 1994 Criminal Justice and Public Order Act.

The reason for the change (AIUI) was due to the arrest of IRA dissidents who remained silent and then produced alibis in court that would have been easy to dispel at the time but were not so easily dismissed in court, once they had gathered support from their communities.

Incidentally, the 1994 act also banned music containing "repetitive beats" and legalised anal sex between heterosexual couples, albeit with a number of caveats and conditions.

More info is available at the Wikipedia page: https://en.wikipedia.org/wiki/Criminal_Justice_and_Public_Or...

andyjpb··on Say it's 1962, you're lucky enough to be a programmer working with an IBM 7090
> The 7090 ran at 458.7 kHz -- https://twitter.com/Foone/status/1201977408175235072

That's actually really fast for 1962.

By 1980 the workstation market was thinking about 3M: A megabyte of memory (RAM), a megapixel display and a million instructions per second (all for less than a "megapenny"). -- https://en.wikipedia.org/wiki/3M_computer

OK, the machines of 1980 we're desktop sized rather than room sized, but, as Foone points out, the 7090 was also really a single-user (at a time) machine.

These days you can get a similar spec (within approximately an order of magnitude) in an Arduino: 8,000 or 16,000 kHz with 32kB (rather than the 7090's 144kB) of RAM.

Moore's law is strong but the applications are varied and sometimes the form factor is what changes the game rather than the raw specs.

andyjpb··on The Spaceline: a space elevator alternative achievable with current technology
My favourite space elevator alternative is the space fountain ( https://en.wikipedia.org/wiki/Space_fountain ).

It's an active structure made of small parts that are held together with a stream of magnets.

If the fountain fails then each part can have its own parachute so that it can safely return to Earth. i.e. without destroying entire cities as would be the case for a traditional elevator design.

andyjpb··on Turbo, an Improved Rainbow Colormap for Visualization
NASA have a cool 6 part series on how to use colour for visualisations here: https://earthobservatory.nasa.gov/blogs/elegantfigures/2013/...
andyjpb··on IBM Open-Sources Power Chip Instruction Set
Google have been experimenting with POWER in their datacenters for a while now: https://www.forbes.com/sites/patrickmoorhead/2018/03/19/head...

More historically, Google have been building their own networking gear for some time https://www.wired.com/2015/06/google-reveals-secret-gear-con...

I'm focussing on Google in particular because they have always had a strong preference for Open components wherever possible and they've traditionally taken advantage that openness wherever they think they need to even if that goes against common practice. (There's a story I can't find the link to where, in the very early days, they wrote their own patches to Linux to work around some bad RAM chips that they'd scavenged from somewhere.)

If Google can get an advantage then they will take it. They will also invest heavily, over years, to research these advantages and opportunities.

Their attitude to things like ARM is still fairly accurate at the scale of their datacenters: https://research.google.com/pubs/archive/36448.pdf

andyjpb··on IBM Open-Sources Power Chip Instruction Set
An open, high end CPU design is really going to change the cloud market. An ISA like this is a first step in that direction.

Facebook and Google already have their own compute projects and, like Amazon, have access to custom versions of silicon from a variety of vendors.

With a properly open CPU design we'll start to see the first tightly integrated, vertical "cloud" products that maybe still have a "commodity" API on the top (or maybe not?) but are custom all the way down from there.

With the end of Dennard Scaling, if not Moore's Law, Open ISAs and Open CPU designs will radically change both the hardware and compute markets and ecosystems over the next 5 to 15 years, similar to what we saw with Open Source in the 1990s.

Of course, it's not clear that POWER will be the one to do that, and RISC-V isn't going to be making a grab for Intel's crown any time soon, but this looks like IBMs bid to lead in that area.

When the cloud vendors start building systems like this they'll not look too much different from mainframes and IBM wants to continue to own that market.

andyjpb··on Swap, swap, swap, and bad places to work
It's still not as simple as that.

Swap is there to relieve memory allocation pressure. Memory allocation pressure is an incredibly dense concept but it basically means "if lots of people are asking for fresh pages, how quickly can I service them"?

There are also other types of memory pressure. One is "if lots of people are reading and writing to pages, how quickly can I service them?"

This depends on the state of the mapping for the virtual page being read or written. If that virtual page has an associated physical page then the answer is "not too slowly". If the physical page is in one of the caches then the answer is "more quickly". If the relevant part of the page is in a register then the answer is "in one clock cycle".

On the other hand, if the read or write is associated with a page that needs to be mapped in, either from a disk file or a swap file, then the answer is "slowly".

These types of pressure need to be balanced with each other. The idea isn't just to keep as much data in physical RAM as possible. The idea is to use the RAM as effectively as possible (perhaps by keeping as much relevant data in RAM as possible) whilst also being able to respond effectively to requests for fresh pages.

In general, under pressure and load, the Linux kernel tries to keep a few MiB of ready-to-allocate pages. When it runs out of these it raids the page cache. It's reasonably quick to get (non-dirty) pages from the page cache because they can just be zero'd and handed out. The most difficult pages to reclaim are those that must be copied to swap before they are zero'd. So the kernel tries to minimise that.

How does it do that? Well, it has some tricks:

When processes load and run they allocate a bunch of pages. Sometimes, and not infrequently, they allocate and write to pages which are never accessed again!

I see this on my laptop all the time. I never use even close to the physical amount of RAM I have in the machine. Right now I'm using 4,959MiB of 15,799MiB and it rarely surpasses 6 or 7 GiB. However, if I leave it running for any length of time (a number of days or weeks) then I start to see a little bit of swap getting allocated. I've currently been up about 98 days and right now I'm using 317MiB of swap.

What's happened here is that the kernel has swapped out pages that it thinks are never going to be used again. That way, if memory allocation pressure suddenly increases it's got those physical pages ready to service those requests.

If there was no swap, those pages would be unnecessarily pinned into physical memory even tho' they would never be used.

Another commenter asked a question like "What's the difference between a machine with some memory and some swap and a machine with more memory?" Well, there are a few subtle differences, but this is one of them. The machine with swap will have a higher percentage of physical memory available and will be able to respond faster to larger allocations.

Another difference is price. RAM is still expensive compared to disk. If you have a 16GiB box with a bit of swap then you have a 16GiB box that you can use for data that's actively being read and written. If you don't have the swap then you're paying for some of that RAM that gets written and never used. This doesn't matter so much in the small scale but when you have a few machines you want to be getting the most out of them (and what that really means is the subject of another post!).

Swap space gets you more bang for your buck.

So swap space is really about giving the kernel a mechanism to manage the different types of memory pressure without taking too many compromises on the different trade-offs. Even if you're never using all your RAM, swap will still get used so that the machine can respond optimally to as many types of memory activity as possible.

One thing I'd really like to know is whether the number `free` gives me for "Swap used" is the amount of data that's in swap and nowhere else or whether it's the amount of swap space that's used even if the pages are also still in physical memory.

andyjpb··on MPEG1 Single file C library
Buffering a several second chunk allows the transmitter to include forward error correction (FEC).

This allows operators to trade off latency for stream integrity which is important when it's going to be transmitted over a lossy medium such as terrestrial broadcast.

This differs from the size problem because the amount of FEC (if any) is a parameter that can be configured between encoder and decoder. A size header would be a built in part of the protocol: hence my assertion that it would force a frame of latency.

i.e. FEC is an operational concern whereas the size header is a protocol design concern.

andyjpb··on MPEG1 Single file C library
Even if you had enough RAM, storing the frame at the encoder just to calculate its length would force a frame of latency at the encoder.

The way it is, the decoder has more flexibility in implementation.

andyjpb··on MPEG1 Single file C library
jsmpeg is by the same author!
andyjpb··on Syslog: Complete System Administrator Guide
For the people down voting this post, please see https://xkcd.com/1053/

The formula is like that because there are 3 bits of information in the Severity Level field: it has 8 values (0->7), and 2^3 = 8.

Multiplying the Facility Number by 8 (or, equivalently, by 2^3 or by left shifting the number by 3), frees up the bottom 3 bits.

This is the same operation as you are suggesting to do in base-10: multiply by 10^3 to free up the bottom 3 digits, but for base-2 (i.e. in binary).

When the Facility Number has been moved out of the bottom 3 bits you can add in the Severity Level and get a combined value for both.

It's true that it's not very intuitive when they then render the resulting number in base-10 / decimal, however, it does get you the most compact encoding given the size of the two fields.

As another poster says, if the number had been rendered in octal (base-8) then the Severity Level would be the right-most number in the series.

andyjpb··on Ask HN: Best Lisp for software development?
I built

Knodium ( Demo video: https://www.youtube.com/watch?v=gOPuWi-dbQg ; Technical video: https://media.ccc.de/v/c116_lisp_-_2013-08-25_11:15_-_buildi... (sound comes in after the first couple of minutes))

and

Registers.app ( https://registers.app )

in Chicken ( http://call-cc.org/ )

...and would recommend it.

The community in #chicken on freenode is really quite something. They're happy to help with all sorts of problems, even basic ones and get stuck in over and above expectations when problems get tough and technical.

andyjpb··on Why Three Prongs? (1996)
It applies to AC.

However, RF is more complicated. The difference between "RF" and "AC" is a matter of what materials and frequencies are involved and how you are using them.

Terrestrial RF systems rely on the Earth as a ground conductor. It becomes more complicated because the impedance (like an AC version of resistance <waves hands>) of a material (such as your body between the antenna and ground) will change depending on frequency. Once a circuit it in place, the impedance mismatch between your finger and the thing it is touching will cause energy to build up at that interface.

This is similar to how light diffracts when it moves from one material to another and you can see the boundary because some energy builds up there and is scattered.

You won't get a burn if there is no circuit.

...but if you are fully enclosed in an RF field, you may get little circuits forming because your body is in contact with that field, and therefore referenced to it, and the potential of the field might vary across your body.

andyjpb··on Why Three Prongs? (1996)
"Normal breaker connected outlets used in US houses" are already connected to ground on one side. If you grab the neutral side, you will (ideally) get no shock. If you grab the phase side then you'll complete the path to the other side, through ground.

An isolating transformer is a 1:1 transformer where the input can be referenced one way and the output another. The output is not connected to ground on either side so when you grab one of the sides, that's the first time it gets referenced to ground. ...and then the explanation continues as per my previous post.

andyjpb··on Why Three Prongs? (1996)
It's not very intuitive, but it is true!

When you have a power source it needs two wires to make a current flow: a wire "out" and a wire "back". You're only part of the circuit if you're connected to both sides.

When the two wires are not connected to anything, there is infinite resistance between them.

If you grab one side then there's no path through you to the other side so no current flows: the resistance between the two wires is still infinite.

Due to Ohm's law, (V=IR), if the current through you is zero then the voltage across you is also zero. The hand touching the wire is at the same voltage as the rest of your body.

So you "float up" to whatever voltage the wire you are touching is at.

If you complete the circuit then current starts to flow. The shock you receive is related to the current that flows through you. However, even tiny currents cause a big shock because humans are quite sensitive to currents in the micro or milliamps range as that's more than our nervous system uses to move our muscles.

andyjpb··on Readable Lisp S-expressions Project (2013)
Something similar to sxml? https://en.wikipedia.org/wiki/SXML

XML has attributes which S-exprs don't. This can be solved by convention (as SXML does) and doesn't really represent a difference in expressibility.

S-exprs have quote and unquote which XML doesn't. This is a fundamental difference and makes it hard to convert Lisp code that mixes code and data representations.

andyjpb··on Dialog – A domain-specific language for creating works of interactive fiction
http://www.linusakesson.net/games/tethered/index.php
andyjpb··on De-Googling My Life
The Gemini ( https://www.indiegogo.com/projects/gemini-pda-android-linux-... ) and the Cosmo ( https://www.indiegogo.com/projects/cosmo-communicator#/ ) are both nearly (but not quite) there.

I have a Gemini and have been very pleased with it and have supported the Cosmo too because I want to support them. I'm hoping that if they continue to make sales they'll continue to develop newer devices.

These projects aren't quite delivering your complete feature set but they're making the best go of it out of anyone I've seen.

andyjpb··on De-Googling My Life
You don't have to be a GMail user for Google to end up with a copy of a large percentage of your eMail.

Because lots of users do have GMail a lot of your messages will end up on their servers.

However, that's not to say it's not a good idea to practice good hygiene yourself and find a provider that suits your goals.

andyjpb··on De-Googling My Life
For Hangouts replacements there are quite a few WebRTC video conferencing sites that can work with URL sharing and either with or without accounts.

I regularly use https://meet.jit.si https://appear.in/ and https://appear.in/

andyjpb··on Common Lisp: The Untold Story (2012)
In the document he mentions that the eMail latency between EU/Japan and US was 2-3 days because of the number of uucp hops that the mail needed to make. He cites this as a reason why it was impractical to collaborate.

Perhaps those issues persisted and once they were gone the culture was embedded?

andyjpb··on Facebook are 'morally bankrupt liars' says New Zealand's privacy commissioner
The thing that makes Facebook different from the postal service is that Facebook provides an amplifier.

You can send one message and it'll be received by many.

Facebook may not have to take responsibility for users' content or wrangling with issues such as free speech but they must take responsibility for what and how they choose to amplify and recommend that content to others.

Traditionally this is a power that has only been held by broadcasters. Broadcasters have things like time delays and "dump switches" when they are conducting live shows. The time delays are mostly used to insert the "bleeps" for bad language. The "dump switch" is a more brutal approach that allows them to avoid broadcasting something that is going terribly wrong.

Broadcasters also have "watersheds" that place time constraints on when certain types of things can be broadcast and broadly define who the expected audiences are.

In this case, Facebook are acting more like a broadcaster than a publisher or message conveyer. Because the audience can be more tightly controlled than a regular broadcaster, there is a case to be made for making the responsibility bar higher, not lower, so that inappropriate content cannot be deliberately targeted at vulnerable people.

andyjpb··on A small trail through the Linux kernel (2001)
That's great!

If you're interested in more detail about how a VFS (Virtual File System; the bit where it finds the file and then finds the data in the file) works, Chapter 17 (Another Level of Indirection) in Beautiful Code (http://shop.oreilly.com/product/9780596510046.do ) is well worth a read.

It explains how the VFS in FreeBSD works. How disks, partitions, volumes, filesystems, offsets, etc all get orchestrated together so that the Unix "single directory hierarchy" abstraction works as intended even tho' the actual data may be spread around a number of locations (both local and remote).

The concepts and ideas are largely applicable to other Unixes and Unix-like operating systems.

It's also a great example of how elegantly one can express polymorphic code, even in plain C.

andyjpb··on Serenity: x86 Unix-like operating system for IBM PC-compatibles
I've seen it done (multiple times).

I worked for a set top box manufacturer where the lead engineer wrote an OS and a GUI system, from scratch, for our platform in less than a year. It had a bunch of syscall personalities. One for vxworks so we could port stuff from our existing platform and one for Linux so that people could write drivers for it based on the O'Reilly Linux device drivers book (which was current at the time). It also had a couple of others for odd things that we needed. It was incredibly well written. It only took me a couple of weeks to port it to PowerPC and the first week of that was just phaffing getting my cross compiler working properly.

In contrast, we had another couple of programmers trying to write a web browser for that platform. After a year they had something that could render primitive web pages but there was still a good chunk of the web that didn't work "quite right". After two years they were still in a similar position.

Operating systems are a dark art, and the person who wrote Serenity is very talented, but OSes are, at least, a tractable and well defined problem.

andyjpb··on SPARCbook 3000ST
Yeah... It makes more sense for /var and even /usr/local. / and /usr are both managed by the OS installation. /opt makes a little more sense because it's often used by large, third party packages any one of which could be larger than the whole of / and /usr together.

If / or /var run out of space (even now) then lots of daemons get into quite a bit of trouble. Anything from hanging to consuming a lot of CPU in the disk allocator.

It's often not possible to ssh into a machine where / or /var has filled up.

There was also the problem of file system robustness. Whilst things were a lot better than the non-UNIX platforms of the day, things weren't quite as good as they are today. These filesystems often were not journalled. That meant that if you had a power failure you could lose the entire volume. (I've personally had at least one / partition be destroyed by fsck after a power failure. I was so, so glad I didn't lose /home. /opt and /usr/local as well!)

Depending on whether you were the user or the administrator dictates which partition you prefer to survive, but at least it adds some robustness. It's nice to have things separated based on how you will restore them. / and /usr will come from the vendor. /opt will probably be a whole load of different media from all over the place.

andyjpb··on Terminal latency (2017)
He was on MacOS so unfortunately couldn't do xterm.

They do uxterm (which is a wrapper around xterm to put it into UTF-8 mode) here.

https://lwn.net/Articles/751763/

TL;DR: This quote sums it up:

"The first thing that struck me is that old programs like xterm and mlterm have the best response time, having worse case latency (2.4ms) better than the best case for all other terminals (10.6ms for st)."

andyjpb··on What Every Programmer Should Know About Memory (2007) [pdf]
Having a certain amount of Mechanical Sympathy (https://mechanical-sympathy.blogspot.com/2011/07/why-mechani... ) is important for having a good intuitive feel for the appropriate way to implement something.

For example, an O(n^2) algorithm can often beat an O(n) one when n is "small". How do you get that "experienced" feel for which to choose when you're designing your program?

This is even important for people doing REST APIs, Web Services and Microservices. Milliseconds matter when the load gets beyond a certain point and your system ends up full of stragglers. How do you get that "experienced" feel for where that point might be and how far you can push a particular architecture without investing in a major rework?

The effect of tail latency on most web servers might surprise you:

http://latencytipoftheday.blogspot.com/

http://latencytipoftheday.blogspot.com/2014/06/latencytipoft...

TL;DR: most of your visitors will experience the bottom 1% of your performance curve on every page load.

← PreviousPage 3 of 5Next →