dottrap

Changes from Lua 5.3

- new generational mode for garbage collection

- to-be-closed variables

- const variables

- userdata can have multiple user values

- new implementation for math.random

- warning system

- debug information about function arguments and returns

- new semantics for the integer 'for' loop

- optional 'init' argument to 'string.gmatch'

- new functions 'lua_resetthread' and 'coroutine.close'

- coersions string-to-number moved to the string library

- allocation function allowed to fail when shrinking a memory block

- new format '%p' in 'string.format'

- utf8 library accepts codepoints up to 2^31

Prior HN discussion on Lua 5.4.0 (beta):

https://news.ycombinator.com/item?id=21145676

Contents:

Lua 5.4.0 (beta-rc2) remains available for testing at http://www.lua.org/work/

Please take a look before we freeze Lua 5.4.0 (beta), which will happen next week if no glitches are reported.

All feedback is welcome. Thanks.

I want to know, how do Costco's sale prices (coupon book) affect the markup/margins on a product? If the normal markup is less than 11%, is the sale price markup less than say 5%?

Edit: Or alternatively, are Costco sales negotiated ahead of time with the product makers so the makers are taking the cut and Costco's margins remain?

And as another example, Mac OS X (10), which debuted in 2000, has had every subsequent version continued to be named 10.X. Just this week, macOS 10.15 Catalina was released.

Funny that the name Mac OS X went to OS X and then (back) to macOS, but the 10.X name persists.

> I wonder if Lua developers have learnt anything from him with respect to performance.

It cannot be emphasized enough that PUC-Lua and LuaJIT have different goals. One goal for PUC-Lua is that it must be portable everywhere and be implementable in pure ANSI C (no platform or architecture specific calls). LuaJIT in contrast contains much handwritten assembly for the specific architectures it supports.

PUC-Lua also cares deeply about interoperating with C and hence why the Lua-C interface is formally part of the language spec, and as the parent thread mentioned, LuaJIT using the traditional Lua-C interface can be really slow.

That said, PUC-Lua has learned a lot from Mike Pall. It was Pall that helped them get fully yieldable coroutines across pcall boundaries. I'm sure Pall's lengthy analysis of unimpressive garbage collection performance in both Lua 5.1 and LuaJIT was read by all of them and a motivation for them to try the generational garbage collector now finally working in 5.4.

But finally, there is a new project called Pallene (not being pedantic, formerly known as Titan). It is inspired by all the lessons of LuaJIT. The name 'Pallene' is a nod to Mike Pall. Pallene is a statically typed sister language to Lua that is designed for performance and easy C/Lua bridging, but in a way that doesn't bring in all the downsides of JIT. The language is glued at the hip to Lua so there is also easy interoperability between Pallene and Lua code. Pallene introduces static types not for the sake of type safety, but for the purpose of inferring what things can be optimized. Pallene brings in a AOT compiler into the mix and generates optimized binaries. But these binaries also look like any other Lua/C native library so they are callable from normal Lua too and the user doesn't know if they were implemented in C or Pallene.

Pallene was previously discussed on HN here:

https://news.ycombinator.com/item?id=18038619

Also look for the original Titan talk from one of the Lua Workshops a few years ago.

I haven't tried them yet, but in the work reference manual, look at: 3.3.7 – Local Declarations and 3.3.8 – To-be-closed Variables

I think the syntax use is: local x <const> = 5 local filename <const> = "/etc/fstab" local fh <close> = io.open(filename, "r")

Main changes

- new generational mode for garbage collection

- to-be-closed variables

- const variables

- userdata can have multiple user values

- new implementation for math.random

- warning system

- debug information about function arguments and returns

- new semantics for the integer 'for' loop

- optional 'init' argument to 'string.gmatch'

- new functions 'lua_resetthread' and 'coroutine.close'

- coersions string-to-number moved to the string library

- allocation function allowed to fail when shrinking a memory block

- new format '%p' in 'string.format'

- utf8 library accepts codepoints up to 2^31

I first read about the style of not typedef'ing structs in the Linux Kernel style guidelines. I read some of the justifications and the pointer issue you also bring up is a really convincing one. But I was also convinced by the context argument. I've seen Linus use this one to talk about the downsides of using C++, but it also applies here. Basically, when reviewing a small block of code, it is typically possible to understand what the code does in C (whereas in C++, because of all the language features that kick in automatic implicit behaviors, it is much harder to know if any of that is going on without much more context.) Similarly with the explicitness of struct, you know exactly what is going on and you don't have the ambiguity of whether the type is a struct, pointer, or primitive type (like int) and how they impact the behavior slightly differently in the language.

Very nice article!

Minor nit: > there’s now a standardized bool with true/false (also not built-in but defined in stdbool.h)

C99 introduced _Bool, which is a built-in.

stdbool.h defines bool as an alias to it (unless compiling as C++, which then gets left alone as the native C++ bool).

A lot of bottled water comes from municipal water plants (aka tap water).

That Penn & Teller YouTube link even has a section showing this, though you only need to go to the supermarket and look closely at the fine print of all the different brands of bottle water. You'll quickly find a lot are tap water.

Any that allows static linking will get the same benefit.

For the three mentioned in the ancestor thread:

Apple doesn't ship a static libc, so developers must dynamically link.

glibc does allow static linking, but it is something they don't really go out of their way to promote and I recall some historic pains trying to get it to work right. Maybe that all works better today.

MSVC does allow static linking to msvcrt. MSVC is a little different than other platforms because they have permutations (and associated rules) you must be aware of: [debug | release], [static | dynamic], [single-threaded | multithreaded].

> And what value what that be?

A lot of embedded development still have very tight requirements. We're still talking about things discussed in tens or if generous, hundreds of kilobytes.

As a more mainstream example you might have heard of, the Arduino, I think the baseline Arduino has around 32KB available.

Also, in the embedded space, the compilers used are notorious for not having a complete standard library implementation. Lots of things are missing. Thus, developers who care about porting their code to different chips, are cautious about limiting what standard library functions they depend on.

Again, as a more mainstream example you might have heard of, look at Lua, which is implemented in 100% ANSI C. Because they care about Lua truly being portable to everywhere, they try to constrain their dependence on the standard library, and they supply some #ifdefs in luaconf.h to implement or workaround some standard library functions they know to be often missing.

And as a more mainstream example of an incomplete C standard library, look at Android. Bionic is the name of their C standard library. It is missing a lot of things. (And you have to use one of those #ifdefs in luaconf.h to compile Lua successfully on Android to workaround Bionic.)

Here is a web page comparing the different C library implementations:

http://www.etalabs.net/compare_libcs.html

For the .a size:

musl: 426k

uClibc: 500k

dietlibc: 120k

If you statically link to these, your final binary will only pull what is actually needed and be smaller than this.

I was curious about that comment so I Googled it. Here is the top hit which has fairly good depth:

https://sanfrancisco.cbslocal.com/2016/05/14/shoplifting-cal...

The summary seems to be that Prop 47 treats shoplifting under $950 of value as a misdemeanor. As a result, law enforcement is unlikely to pursue and punish, so thieves get away scot-free.

Additionally, organized crime has exploited this and can easily recruit people for their shoplifting rings. The article reports that retailers have seen a rise in shoplifting since the law took effect.

So presumably, package theft of anything below the value of $950 from a door step (not a federal USPS mailbox) would only be a misdemeanor in California.

Dave Ramsey recommends Churchill Mortgage.

Dave Ramsey has encouraged millions of his followers to do the same ("indeterminable" score"). For the most part, it is not that much of a problem. Even home mortgages can done through manual underwriting.

The biggest pain point seems to be car rentals as they seem to have constantly shifting and strange rules about whether and when they accept debit cards. The next seems to be car insurance discounts as insurers seem to have strangely connected your driving risk with your credit score.

They take a cut from every transaction you make. They make money even when you always pay off your balances. Your benefits were paid for by the transaction fees absorbed by merchants.

So if you want to take a stand, use only cash.

There was a topic a few months ago about running JavaScript code in ring 0. People used similar arguments to claim it was safe enough. A v8 engineer chimed in to explain the reality of JavaScript engines.

https://news.ycombinator.com/item?id=17186842

JavaScriptCore has a huge surface area. It is measured in tens of megabytes, whereas Lua is measured in hundreds of kilobytes. Wikipedia chose Lua because it was small enough that they were able to audit every line of code for security. Verisign approached Lua the same way for their high-reliability servers.

Consider the complexity of JavaScriptCore. Their new-ish "FTL" optimization engine stands for "Fourth Tier LLVM", which means there are at least 4 levels of runtime optimizations JSCore performs on code, which is a lot of non-trivial complexity. And JIT means they cannot utilize no-execute bits in hardware.

Then there is the memory and power-consumption. Lua's footprint for a VM instance is tiny. It is also a very efficient language, even in interpreted form. JavaScriptCore is comparatively heavy in both regards.

Remember that Apple Watch still does not have Safari and JavaScriptCore is not available for watchOS developers. Presumably this decision is partly influenced by memory and performance constraints. Lua is trivial to get working well on watchOS.

For a core feature, it makes sense to have minimal dependencies with a minimal footprint so plenty of room is left for user applications. Remember that Apple probably has a bunch of skunkworks devices in their labs they hope to productize someday. And many of these might be tiny portable devices with minimal hardware capabilities, like Apple Watch (or remember all the different iPods?).

Lua's tiny implementation and extreme portability (written in pure ANSI C) make it well suited for this role. Apple won't have to port Lua to every new experimental CPU architecture they try, whereas JavaScriptCore has a lot of platform specific #ifdefs which have to be dealt with for every new platform.

I use Shrook. It is native, lightweight, fast, and free.

> Does the Mac App Store require submission as source code?

No. You do not submit source code.

> They require (I think -- maybe just strongly encourage) submissions to the iOS App Store to include LLVM bitcode.

Apple does not currently require LLVM bitcode for iOS. It does require it for Apple TV and Apple Watch.

Additionally, LLVM bitcode is not architecture agnostic. It would not solve the problem of going from x86 to ARM.

> (anyone remember fat binaries i.e. the ones including both PPC and Intel executables in one .app container?)

This was also used in Apple's transition from 32-bit to 64-bit on Mac. And later again on iOS.

So yes, if Apple were to transition CPU architectures again, fat binaries would be the likely approach.

I recently was fixing a relative's early gen Macbook Pro which had the same keyboard as the Powerbook. The keyboard felt soooo nice. I really miss that keyboard.

Here's another link which compares musl, uClibc, dietlibc, and glibc sizes, features, and performance.

http://www.etalabs.net/compare_libcs.html

It says musl's smallest static C program can be 1.8k as opposed to glibc's 662k, showing libc doesn't always have to be big and bloated.

You can set a count hook to defend against infinite loops.

Lua is great.

> More expensive to start than threads,

Maybe, but, on Linux, processes and threads are almost the same thing.

Additionally, even where a process is a bit more expensive to create, it is not enough to block the UI thread from being responsive. I have first hand experience with this on different operating systems, including Windows, and it is more than fast enough to keep the UI completely responsive.

> and far more expensive and complex and restrictive to move data around.

Not necessarily. For threading, synchronization patterns are not necessarily simple. (This is why computer science instruction spend time on these principles.)

Furthermore, some languages and frameworks provide really nice IPC mechanisms. Apple's new XPC frameworks are pretty nice and make it pretty easy to do.

> Either the tasks aee broken into ridiculously fine-grained bits that are hard to make sense or keep track,or you still get a blocking UI.

As I mentioned, coroutines make this dirt easy. It principle, this doesn't have to be hard.

> Furthermore, the solution is computationally more expensive.

That doesn't really follow. The underlying task is the where the computation is. You are just moving it, either to a process, a thread, or dividing it up, or something else (e.g. send it to a server to process). At the end of the day, it is the same work, just moved.

Yes, you might need some state flags for breaking up the work, but threading also requires resources such as creating and running the thread, the locks and protecting your shared data, and so forth. There is no free lunch any way you do this.

I don't claim these are "go-to" solutions, but only that there are multiple solutions to pick from.

One solution is processes (mentioned in the post). Fork a process which does your computationally expensive thing and then get the result when you are done. For the security minded, we've seen this make a bit of a come back because separate processes can be run with more restrictions and can crash without corrupting the caller. We see this in things like Chrome where the browser, renderers, and plugins are split up into separate processes. And many of Apple's frameworks have been refactored under the hood to use separate processes to try to further fortify the OS against exploits.

Another solution is break up the work and processing in increments. For example, rather than trying to load a data file in one shot, read a fraction of the bytes, then on the next event loop, read some more. Repeat until done. This can work with both async (like in Javascript) or you can do a poll model. Additionally, if you have coroutines (like in Lua), they are great for this because each coroutine has its own encapsulated state so you don't have to manually track how far along you are in your execution state.

For C vectors, look at stb's[1] stretchy_buffer[2] or klib's[3] kvec[4].

The basic concept of these is they use macros to specify the type/size of the elements so you can have generic containers in C.

For hash's, you can also look at klib's khash [5]

[1] https://github.com/nothings/stb

[2] https://github.com/attractivechaos/klib

[3] https://github.com/nothings/stb/blob/master/stretchy_buffer....

[4] https://github.com/attractivechaos/klib/blob/master/kvec.h

[5] http://attractivechaos.github.io/klib/#Khash%3A%20generic%20...

In my above response, it was implicit, but not explicit, that because libraries may use a different memory allocator than your main code, well constructed libraries provide their own Create/Destroy pairs. So to be explicit, you should really never call free() on memory allocated within some library, and that library should provide its own free function. (Windows devs might also remember that crossing DLL boundaries exposes this same issue, so calling free in your program from memory allocated in the DLL is potential hazard.)

I also wanted to state that "information hiding" is typical for C libraries, because it is often useful for maintaining ABI stability while allowing for future changes.

For example, SDL is very careful about keeping the ABI stable for the life of a series. They broke it between SDL 1.2 and SDL 2.0, but that was a long number of years in between.

If they change a public struct that you use, like SDL_Rect, all the memory layouts in that struct may change. That means if your program is dynamically linked, and the SDL that gets used is newer than what you compiled against (and somebody mucked with SDL_Rect), your program is not going to work correctly.

If everything is behind an opaque pointer, then they can change the internals without breaking your binary. So as an example, a running joke is that Apple seems to change the way they do fullscreen every macOS release. This means your game won't work correctly on the next macOS. So SDL will need to add new code to support whatever the new thing is, while still preserving the old code for the existing macOS versions. This could mean a lot of internal changes to SDL's structs. But since they were opaque to your program, it doesn't affect the ABI. So you can just drop in the latest SDL dynamic library with your program (no recompile necessary), and things will work.

SDL_Create/SDL_Destroy basically malloc/free under the hood. (SDL has its own custom allocator, so depending on your platform and compile switches, it may use something other than the standard library malloc/free.)

SDL has some value struct types like SDL_Rect which APIs sometimes pass around through pointers. SDL_RenderCopy is one example.

I don't claim to have any answers, but I found all of this interesting and surprising. I wondered about a couple of things. What happens if the do_stuff_to_foo is actually defined (and what happens in that actual function)? And is there a difference between value semantics and pointer/reference semantics?

These questions were my take-away from one of Chandler Carruth's C++ compiler optimization talks, I think it was this talk. https://youtu.be/eR34r7HOU14

My take aways were that the optimizer gets a huge chunk of its performance by inlining. And with value semantics, the optimizer can "cheat like crazy".

So I defined two different variants for do_stuff_to_foo

Original Pointer/Reference Semantics:

    void do_stuff_to_foo(foo* a)
    {
        a->x++;
    }

Value Semantics:

    foo do_stuff_to_foo(foo a)
    {
        a.x++;
        return a;
    }

In both cases, the compiler emits effectively the same output for C and C++ (I only tested clang.) (The main difference was name mangling. I omit stuff for brevity.)

Pointer/Reference Semantics:

    Lcfi2:
        .cfi_def_cfa_register %rbp
        incl	(%rdi)
        popq	%rbp
        retq
        .cfi_endproc
                                        ## -- End function
        .globl	_blah                   ## -- Begin function blah
        .p2align	4, 0x90
    _blah:                                  ## @blah
        .cfi_startproc
    ## BB#0:
        pushq	%rbp
    Lcfi3:
        .cfi_def_cfa_offset 16
    Lcfi4:
        .cfi_offset %rbp, -16
        movq	%rsp, %rbp
    Lcfi5:
        .cfi_def_cfa_register %rbp
        movq	%rdi, %rax
        popq	%rbp
        retq
        .cfi_endproc

Value Semantics:

    Lcfi3:
        .cfi_offset %rbx, -24
        movq	%rdi, %rbx
        incl	16(%rbp)
        leaq	16(%rbp), %rsi
        movl	$1036, %edx             ## imm = 0x40C
        callq	_memcpy
        movq	%rbx, %rax
        addq	$8, %rsp
        popq	%rbx
        popq	%rbp
        retq
        .cfi_endproc

What I find interesting here is that in both the C and C++ , the memcpy now appears. And in the C case, there is still only one memcpy, not two.

So as I said, I don't have any answers and really don't know what the take away is. But RVO no longer seems to be a factor in these variants.

I've never had a reason to suspect CMake's Makefiles are slower than others. I've converted projects from Autotools to CMake and CMake was always miles ahead faster.

I recently switched to using CMake's Ninja generator. I've been shocked at how much faster it is than Makefiles, even for tiny projects.