HNHacker News
TopNewBestAskShowJobs

Avernar

397 karma · joined November 5, 2014

submissionscomments
Avernar··on How Python does Unicode
If the string is stored as ASCII characters or Unicode code points (UCS-16 or UCS-32) then you are correct that not much changes. But if the string is in UTF-8, UTF-16 or the string system uses graphemes then indexing goes from O(1) to O(N). Every index operation would have to start a linear scan from the beginning of the string to get to the correct spot. With iterators it would be a quick operation to access what it's pointing to and very quick to advance it.

My argument is that iterators are far superior to indexing when using graphemes (or code points stored as UTF-8 but grapheme support is superior). And they don't hurt when used on ASCII or fixed width strings either so the code will work with either string format. No hairs, split or otherwise here.

Avernar··on How Python does Unicode
It didn't have emoji but it did have other combining characters. While some langages it's feasable to normalize them to single code points but other langagues it would not be.

Plus the fact that some visible characters are made up of many graphemes the number of single code points would be huge.

As to your second point it seems to me to be a little close minded. The whole point of a universal character set was that languages can be added to it whether they be textual, symbolic or pictographic.

Avernar··on How Python does Unicode
I don't see why this would be hard with iterators. You have an iterstor to the start of the HICN, either at the start of a or deep in the string. Take a second iterator and set it to the first. Loop six times advancing that iterator checking to see if it's a digit. Then check if the next position is a space.

For the prefix and suffix and how many characters between them you do the above but use the second iterator to find the suffix. Then you either keep track of how many characters you advanced or ask for how many characters between the two.

It's very easy to think about it this way as that's how a normal (non programmer) human would do it. Basically the code literally does what you wrote in english above.

My point being is that iterators are much faster than indexing when the underlying string system uses graphemes. You can do pretty much anyting just as easy or easier with iterators than with indexing. The big exception is fixed width columnar tet files. I've seen a lot of these in financial situations but fortuanately those systems are ASCII based so not an issue.

Avernar··on How Python does Unicode
I didn't realize the modulo operator for bytes was added. most information I've run across said it didn't work.

Unfortunately most libraries for 3 will be using str so using bytes with UTF-8 inside will become more and more difficult.

Avernar··on How Python does Unicode
Look back a few posts. We're discussing using UTF-8 in str and avoiding the unicode type in Python 2.

I'n my use case I validate the string as UTF-8 from the internet. To and from the database is UTF-8 so no validation is required there. Output back to the internet requires no additional steps.

Nowhere in this method is encode or decode required or desired.

Avernar··on How Python does Unicode
> I definitely need indexes

No you don't. You need iterators, which behave like pointers. Let's say you're hundreds or thousands of characters into a string at the start of some token. Now you want to scan from that position to the end of the token.

With indexes it works fast only if it's by codepoint. in a language that properly supports graphemes this would mean it would have to scan from the beginning to get to that index.

With iterators it can start scanning from that position directly. Same speed no matter where you are in the string. With indexes the larger your input the slower your parse gets, and not in a linear way.

It's also super easy to get a slice using a start and end iterator. As for line x character y messages, you can't get that directly from an index as it depends on how many new lines you parsed so indexing doesn't help there.

Avernar··on How Python does Unicode
> But when you get right down to it, what I do for a living is write web applications,

That is my use case for Python as well.

> sometimes I have to write validation that cares about length,

That's where a trucation function that understands grapheme clusters whould come in so handy. Tell it that you want to truncate to n bytes maximum and let it chop a bit more as to not split a grapheme cluster.

Fortunately my database does not have fixed with strings so I rarely bump into this one.

> or about finding specific things in specific positions, and so indexing into a string is something I have do to from time to time

I write my code to avoid this. Yes I still have to use an index because that's what Python supports but it would be trivial to convert it to another language that supports string iterators.

Avernar··on How Python does Unicode
All Python 3 did was put a hard barrier between bytes and strings. That's it.

Mising is all the grapheme handling that languages that do Unicode strings right have.

Avernar··on How Python does Unicode
> Since the high-level API is supposed to let you treat a string as a sequence of code points,

I disagree with that premise. It should operate on grapheme clusters. Operating on code points falls into the same trap as operating on bytes.

> a correct implementation (which Python didn't have until 3.3!) would've imposed the overhead of conversion to something resembling a fixed-width encoding whenever a programmer invoked certain operations.

Those operations should have been removed. Indexing is the big one that needs fixed width internal representation for speed. Code could have been rewritten to not require indexing. But mechanical translation from Python 2 to 3 was a goal and because of that they couldn't radically change the unicode API for the better.

> And the vast majority of strings in real-world Python contain only code points also present in latin-1, which means they can be stored in one byte per code point with this approach. And for strings which can't be stored in one byte per code point, you were similarly going to pay the price sooner or later.

You're going to pay the price for 4 byte per codepoint strings quite often. A single emoji will blow up a latin-1 string to 4 times the size.

Avernar··on How Python does Unicode
> This only works if every library that you use agrees with you on this, and treats all strings you pass to it as UTF-8 whenever encoding matters.

Not exactly. The library just has to treat it as a string and not worry about the encoding (i.e. not try to encode it to/from the unicode type).

Only ran into this issue once and the library had an option to return everything as string so not a problem.

> At least in Python 3, with bytes, if a library does try to use it as a string, you'll get an error, rather than silent wrong output.

Bytes in Python 3 don't support string operators.

Avernar··on How Python does Unicode
What do you mean by "expose UTF-8"? Because nothing about UTF-8 requires that you give byte access to the string.

As for indexing, strings shouldn't require indexing period. That's the ASCII way of thinking, especially fixed width columns and such. You should be thinking relatively. For example, find me the first space then using that point in the string the next character needs to be letter. When you build you're code that way you don't fall for the trap of byte indexing or the performance hit of codepoint indexing (UTF-8) or grapheme indexing (all encodings).

Avernar··on How Python does Unicode
One reason is because it would take a lot more code points to describe all the possible combinations.

Take the country flag emoji. They're actually two seperate code points. The 26 code points used are just special country code letters A to Z. The pair of letters is the country code and shows up as a flag. So just 26 codes to make all the flags in the world. Plus new ones can be added easily without having to add more code points.

Another example is the new skin tone emoji. The new codes are just the colour and are put in front of the existing emoji codes. Existing software just shows the normal coloured emoji but you may see a square box or question mark symbol in front of it.

Avernar··on How Python does Unicode
> I think UTF-8 is generally the wrong way to expose Unicode to a programmer, since it lets them think they can keep that cherished "one byte == one character" assumption right up until something breaks at 2AM on a weekend.

The solution to that is simple, don't let the programmer access individual bytes in a Unicode string.

Get rid of indexing into them and replace it with iterators. Make string handling functions work on code points at the very least but better yet on grapheme clusters. There's a little more to it than that but it's a good start.

Yes, people are still stuck in the ASCII mindset and can't seem to get away from thinking in bytes. But I belive it's the ability to index into strings is what's to blame and not the encoding used.

Avernar··on How Python does Unicode
I'm not a fan of how Python 3 stores Unicode strings internally. In my opinion they should have went with UTF-8. The extra scanning and conversion puts more preassure on the processor and caches under load.

I agree that Python 2's Unicode handling is broken. That's why I just stored UTF-8 in a normal string and avoided the whole mess. The only thing I have to do is validate any input from the outside world is really UTF-8.

Avernar··on Dealing with IPv6 fragmentation in the DNS
I don't see an issue with extension headers in IPv6. The RFCs have rules on who should look at what header and what to do when an unrecognized one is encountered.

Most firewalls I've dealt with have some option to defragment the inbound packet at the firewall so that the rest of the rules can be processed properly and to protect nodes from malicious fragments.

Avernar··on From Julian to Gregorian Calendar
"Switch Took More Than 300 Years"

I sure hope the complete switchover from IPv4 to IPv6 also doesn't end up taking that long.

Avernar··on An OSI model for the 21st century (2014)
> Please point me to a definition of a "real layer 3 switch."

All definitions of layer 3 switch I've found were stuff like this: https://en.wikipedia.org/wiki/LAN_switching#Layer_3_switchin...

Those definitions are parroting the manufacturer's definitions. Those devices are routers with ASIC switching using the IP address.

My definition of a real layer 3 switch is this: A device that forwards packets based on the IP address. The decision on which port to forward is based on where an IP was last seen and not on a configured network/netmask value.

Nobody that I know of sells devices like this but you can build one for yourself. The reason I'm going on about a pure layer 3 switch is because it's on one end of the discussion about these hybrid "Layer 3 Switches" that the network manufacturers are selling. It's like trying to talk about what a mule is when you refuse to define what a donkey is.

So we have a pure layer 3 switch on one end and a layer 3 router on the other. The "Layer 3 Switches" that are now being sold are about 95% of the way towards the router side of that line.

> If you want point-to-point ethernet, put your end nodes on a IPv4 /31 (or /30, depending) and be done with it. You will have a "pure layer 3" environment: all traffic will be routed.

But then you're routing and have different subnets everywhere. You won't be able to transparently replace a layer 2 switch with that.

Ethernet framing is redundant. The LLC/SNAP header has already been deprecated by the IEEE so there's 8 bytes saved. The ethertype field is redundant as the first byte of the payload will tell you if it's IP v4 or v6 so there's an extra 2 bytes. The source and dest MAC addresses are redundant as we have source and dest IP addresses so there's a 12 byte savings. The vlan tag would need to stay for now because of IPv4 but we could replace it by repurposing some of the 0 bits in an IPv6 link local address. The preamble, SFD and FCS would stay although the preamble might be able to be shortened as it's minimum length was dictated partly by how many hubs a packet could cross.

So you replace your layer 2 switch with this layer 3 switch. Everything still works with no configuration necessary. As new network cards that can do IP framing come out you configure each port from ethernet framing to IP framing. So now it's a Layer 3 switch on some ports (Ehternet framing) and a Layer 2 switch on others (IP Framing).

Once all ports are switched over the device has now become a Layer 2 IP switch. Now you can see why I don't like the definition of Layer 2 = Switching, Layer 3 = Routing. The device functions as a switch regardless of the framing on each of the ports.

If you use the more general definition of Switching = inside the same network (i.e. subnet) and Routing = between networks then things become more clear.

I know some people on the internet want IP framing to become a thing. Will it? Maybe, maybe not. But by using the more rigid OSI definition of switching and routing it becomes harder to talk about topics like that.

That's why we have TCP/IP now. Because some people were willing to think outside of the OSI box.

> It sounds like you want to eliminate the broadcast domain

It wouldn't eliminate it. The broadcasts would eventually become either IPv4 broadcasts or IPv6 multicasts. It would reduce the sizes of the domains and reduce traffic however.

Avernar··on An OSI model for the 21st century (2014)
> The only reason googling shows so much confusion is most people don't understand the definitions of routing and switching.

No, it means that people are using a broader definition that is not locked into the OSI model. Both definitions are valid but the OSI one is a subset of the other one. Than means the OSI one can't describe certain network devices that exist in the real world. A layer 3 switch can't exist according to the OSI model. It can in the real world.

Go way back to the parent of this thread. See the sentence "Anyway, OSI model is mostly a mental picture that bear little practical relevance."? Yet you're here arguing we have to use the OSI model's definition of switching and routing.

> "Layer 2" = switching. "Layer 3" = routing.

Only if you think that the OSI model is the be all end all definition of what a network is. It's not. The TCP/IP designers didn't think so either and moved functionality that was traditionally in the lower layers to the upper layers.

> "Layer 3 switch" = a marketing term

Yes, it's been turned into a marketing term. But to understand why it's a marketing term you need to know what a real layer 3 switch is. Then it becomes readily apparent why they really created is a router with a handful of switching features rolled in for speed.

> Your "pure layer 3 switch" that only services a single subnet is actually a regular old layer 2 switch.

No it's not. If you could modify the end nodes in the network (you can with Linux or the *BSDs) to use the broadcast MAC address for the destination and have them ignore the ethertype field you'd completely bypass Layer 2. The layer 3 switches would still be able to function and switch packets to the proper nodes. A layer 2 switch in this situation would still work but would broadcast all packets to all nodes and thus stop acting as a switch.

> It doesn't need to be aware of "layer 3" at all.

If it's not aware of layer 3 at all then it's not a layer 3 switch.

A layer 3 switch by definition needs to understand layer 3. So if you had one that just switches using IP (v4 and/or v6) it would not be able to switch other protocols like IPX for example. A layer 2 switch would switch anything.

Avernar··on An OSI model for the 21st century (2014)
You've completely missed my points and at the same time gave a perfect example of someone who's rigedly trying to follow the OSI model.

> IP operates at layer 3. Layer 3, by definition, implies routing.

Layer 3 does not imply routing! I can shut down my internet link and still use a layer 3 protocol to talk to my file server from my workstation. No routing is happening because the packets are not leaving my network.

> > Their definition of routing or switching is whether it's done in software or hardware.

> No. No, it's not.

I know it's not. That's what the hardware manufactures are saying on their websites. It's stupid but it shows how blurry the line is between switching and routing.

> I think that, in general, you kinda sorta know what you're talking about (although you've got a few things incorrect) but the way you keep screwing up basic terminology really makes me wonder.

I know exactly what I'm talking about and know my terminology. The problem is you're trapped in the OSI model and can't see where things overlap.

> In networking, router, switch, frame, and packet are all clearly defined -- as are the various layers and the PDUs involved at each of them -- and mean specific things but you can call 'em whatever you want to, I suppose.

Routing and switching are not as clearly defined as you think. Just google routing vs switching and you get many variations. Some talk about frames and packets, others about MACs and IPs, but the core principle common to them all is that switching is in network and routing is out of network. A few also mention that routing is a more specific form of switching.

Your issue is you mistakenly belive that routing and switching are each tied to a specific layer in the OSI model. With that belief, yeah, frames are switched and packets are routed. But as I and others here have stated, things in the real world don't always follow the theoretical model.

Take for example the idea seen frequently online of doing away with the ethernet layer and having IP as the frame format. So now in the OSI model what was in layer 3 and 4 now become 2 and 3. Or does some of it drop and some stay in the same spot? It quickly turns into a mess. Mainly because what was supposed to happen in layer 2 (flow control and acknowlegements) is happening in layer 4 (TCP). Mainly because the designers of TCP did not follow the OSI model.

My point is that you can't use what was defined in the OSI model to rigedly define what's happening in real world protocols.

As I said, you can create a switch that operates at layer 3. It would learn what nodes are on each port by watching ARP requests. It would send out ARP replies for nodes on other ports with its own MAC and would also have to forward DHCP requests.

This device would function exactly as a switch. You could have the same subnet on many ports and it would work. You can move a node from one port to another and it would work. A router can't do those things. A router forwards packets based on their destination network.

But if you can't understand that and are stuck in the OSI model, we've esentially taken what was at layer 3 and put it in layer 2. The old layer 2 is just there for comatibility. You could even rip out one of the ethernet links between two of these layer 3 switches and replace it with a serial link that just transmits IP packets and everything would work as normal.

Avernar··on An OSI model for the 21st century (2014)
Again, theory and reality rarely line up neatly.

A switch moves packets between devices inside a network and a router moves packets between networks. Calling one a frame and one a packet is splitting hairs and irrelevant to whether something is a router or a switch.

Using a *BSD/Linux box I can make a L2 router. I can also make an real (non routing) L3 switch. And since "pure layer 3 switch" is my made up term I think I get to define what it means, thank you very much. It's something that doesn't route, or in other words, you can have the same network (i.e. subnet) on multiple ports.

Then we get into the definition of a network. Is it a building, a department, or how about multiple buildings all over the world? Or how about the network of a backbone provider? Their core "routers" are really switches as they pass packets to other devices inside their network.

And the network gear manufacturers are muddying things even more. Their definition of routing or switching is whether it's done in software or hardware. That's why they're calling some of their equipment a Layer 3 Switch. As I said, that really is just a fast router based on the above definition.

The easiest way to distinguish switching vs routing is whether the packets move to the same or different subnet. It's a good rule of thumb but there are always exceptions.

You can take an L2 switch and replace it with an L3 switch on your network and everything IP based will work as if nothing had happened. You'd just get less broadcast traffic.

Avernar··on An OSI model for the 21st century (2014)
Which proves my point. Real world stuff rarely fits neatly into theoretical models/categories.

Real world Layer 3 Switches are a hybrid between a router and a pure layer 3 switch. In fact they're just a really fast router.

A normal edge router switches based on IP and a bunch of extra rules. These extra rules range from firewall stuff to QoS. These extra rules need software to process.

A core router is what is now called a Layer 3 Switch. It just cares about the IP address and because of that can be done in hardware making it really fast. It's still a router because as you said it still has routing tables based on networks and subnets.

I don't think anyone makes a true (non-router) layer 3 switch. The only use case I can think of would be a campus wide WiFi network.

Avernar··on An OSI model for the 21st century (2014)
A Layer 2 switch is something that works on the data link layer. A normal Ethernet switch falls into this category and the switching is done by MAC address.

A layer 3 switch is something that works on the network layer. It switches using the IP address.

A layer 3 switch is not a router. It does not switch based on the subnet/network mask. It just remembers which IP is on which port just like an Ethernet switch remembers which MAC address is on which port.

Avernar··on An OSI model for the 21st century (2014)
When I used to deal with people straight out of school they would always talk about the OSI model and it would drive me crazy. Everything we did would have to be categorized. What layer is this in, what layer is that in. I just want you to plug that Ethernet cable into that switch port and configure the static IP address in the software so we can get the hell out of here and have a beer.

No amount of explaining that stuff in the real world doesn't neatly fit into those categories would get them to shut up about it. One guy nearly had a mental breakdown when he first came across a tunnel. "How can you have two network layers?!?"

One day I was installing something in a computer lab at a college and on the wall was a giant poster of the OSI model with all the protocols imaginable listed on it. I had to fight the urge to set it on fire...

Yes, models are great for teaching and for certain types of discussions (L2 vs L3 switches) but the real world is always more complicated and the lines in the model tend to blur. Take the basic computer model taught to new students with the CPU, memory, input, output, storage, networking all neatly connected in that familiar diagram. A real computer is never connected that way. Memory is all over the place, storage can be directly on the CPU or hundreds of feet away in another box, etc.

Students quickly figure out that the computer model is just a teaching tool. Just holding a motherboard quickly makes them see that. But for networking they seem to hang on to the OSI model like if it were a piece of flotsam after a shipwreck.

Avernar··on An OSI model for the 21st century (2014)
The end to end principle pretty much made the flow control and re-transmission functions of the LLC sublayer of the data link layer obsolete. So now we pay a 6 to 8 byte penalty (8 bytes if the MAC can tell us the packet length) on every frame transmitted. Wired/optical Ethernet does not normally use LPD (LLC Protocol Discrimination, i.e. 802.2 LLC/SNAP) but unfortunately WiFi does.

So in 2014 while the author is advocating for the data link layer to ensure packets are received by other devices on the channel like the current OSI model does, the IEEE is encouraging new standards to use EPD (Ethernet Protocol Discrimination) instead of LPD. The IEEE just wants the LLC sublayer to only do multiplexing of protocols now.

The author completely missed how the upper layer protocols, i.e. TCP/IP, has made some parts of the lower layer protocols unnecessary. Having a basic model is good thing for classifying different parts of the network and makes discussions clearer. But in the real world things rarely fall into categories that neatly.

Avernar··on Migrating 1200 databases from MySQL to Postgres
I'm running postgresql in a FreeBSD vnet jail without issue. The network stack is not shared as it's a vnet jail. I have it set for per jail sysvipc, new in FreeBSD 11, which was the last thing I needed to fully isolate it from other jails.

Is there anything else that's needed to properly restrict the jail?

Avernar··on Show HN: Shareable reasons to upgrade to Python 3
I wouldn't mind the strict demarcation if the internal format was UTF-8. Then a simple validate call on input would be all that was needed.
Avernar··on Apple staffers reportedly rebelling against open office plan
That would be nice if people had private offices but it looks like a dozen or so to each enclosed area.

Sounds like (pun intended) that they've made it more like a concert hall. Instead of white noise with all the reflections you get to hear everyone in your jail block, er, work area nice and clearly.

I personally find that hearing a clear conversation beside me to be a bigger distraction as I tend to "lock on" to it instead of thinking of the problem at hand. So your damned either way, loud but more noise like or quieter but more distracting.

Avernar··on The man who wrote the book on password management regrets the error
Nope, it's one bit extra entropy per character. The attacker must try all possible capitilazations of each word (horse, Horse, hOrse, HOrse, etc).

You've doubled the character set so it's not negligable.

Avernar··on Harvesting Cb Response Data Leaks
Software that is designed to prevent malware from uploading sensitive information to the internet has an option to upload sensitive information to the internet. What idiot thought that was a good feature to add regardless of the warnings? It makes the software do the complete opposite of what it's supposed to do.

If someone needed that then it should have been a special version of the software, not sometthing that's a single option to enable. Way to easy for it to get turned on by accident. Even smart people sometimes make mistakes, not to mention interns, managment that likes to pretend they're techies, botched upgrades, etc.

Avernar··on Apple staffers reportedly rebelling against open office plan
Looks like an Apple store. Bet it's going to be as noisy as one.
← PreviousPage 3 of 9Next →