vii

The blog post describes him trying to catch up the deficit from vacation days away from the Internet.

StackOverflow has done a great job with the points incentives. As noted in this blog, the daily cap on points is important - it's valuable to put the brakes on people who are too into it and in danger of burning out. Candy Crush does the same thing.

A few years ago, I asked Joel Spolsky what he might do differently: he mentioned that getting to a million points is a magical number for many people but it's too tough on StackOverflow. If the point system were set up to be more generous, more people could achieve this milestone and more valuable contributors would have this somewhat arbitrary but enjoyable recognition

This is the paradox of interviewing for senior roles of all kinds: the real work is long term over years and needs careful decision making, but the interview is under an hour.

Well structured thought and the ability to communicate and adjust priorities are key skills for PMs, whereas the actual design of the solution, defined by input from experts across disciplines, is less important. Pleasing the interviewer as the main objective may actually be a good way to demonstrate capability for this role of PM (distinct from subject matter expert roles).

This study from Cornell attempts to estimate expenses and total costs in Seattle across both platforms https://digitalcommons.ilr.cornell.edu/cgi/viewcontent.cgi?a...

Drivers received about $23.50/hr according to it.

The Parrot and Reich recommendations https://irle.berkeley.edu/files/2020/07/Parrott-Reich-Seattl... are much more generous in terms of estimating expenses. Neither considers a company car model which might imply drivers taking shifts in one vehicle.

clang also doesn't assume -march=native - just like gcc it makes binaries that can run on machines of the same architecture rather than specialising to the current processor's features.

You can see the flags it's using by comparing

   clang -E -v - </dev/null 2>&1

and

   clang -march=native -E -v - </dev/null 2>&1

The second output will probably show a lot of important features enabled - for example

   +sse4.2 -target-feature +avx2 -target-feature +fxsr -target-feature -wbnoinvd -target-feature +sse -target-feature +lzcnt -target-feature +pclmul -target-feature -prefetchwt1 -target-feature +f16c -target-feature +ssse3 -target-feature -sgx -target-feature -shstk -target-feature +cmov

The gold linker and now lld can do LTO pretty fast across huge codebases: this breaks down the techniques used https://archive.fosdem.org/2019/schedule/event/llvm_lld/

Curious that Intel seems to recommend dynamic linking to get architecture specific libc function implementations. Dynamic linking has considerable overhead.

py.test has a different way of finding tests. It loads Python files starting with "test_" or ending in "_test.py" then uses reflection, to find functions starting with "test_" or classes starting with "Test", or functions marked with the attribute "__test__" (which is applied by the @nose.tools.istest decorator). See https://github.com/pytest-dev/pytest/blob/d69abff2c7de8bc65b...

This mechanism of scanning the AST is a different approach and won't find the same set of tests. Scanning the AST can be useful and things like https://macropy3.readthedocs.io/en/latest/ are awesome though!

Yes, the network works back from the signed key, which is published last. The TESLA scheme uses a one-way pseudo-random hash function to generate keys, see this paper https://www.esat.kuleuven.be/cosic/publications/article-2749... so if you know the signed key you can generate the others.

That means you can't easily fork a new chain with a K_6' as the receiver can easily check f(K_6') = K_5 where f is a hash function that is hard to reverse.

Once the key has been published, an adversary can use it to generate messages. A requirement in TESLA is that there is close time-synchronisation between the sender and receiver. This is enforced by the safe-packet test as defined in https://tools.ietf.org/html/rfc4082 which checks that the message packet was received within the time window expected from key signing.

As the satellite positioning systems depend heavily on highly synchronised clocks this is pretty reasonable.

Assuming you're doing bulk inserts, and can restart the insert on any failure, one tactic is to disable all consistency and durability storage features during the insert. This heavily reduces the IO operations needed and the need to wait for them to complete. That's assuming that iops are the resource that you are constrained on.

Steps:

- use ext2 or even tmpfs as the filesystem, this disables journaling and COW features of ZFS; mount options async,nobarrier

- set tables to be UNLOGGED to disable Postgres WAL

This got things fast enough for my last use-case. But next ideas, as at some point you then become CPU/memory bound

- you could try sharding by partitioning the table

- another trick is to use SQLite as a backing store for inserts (it is quite fast if you turn off all logging) and then query with Postgres via a FDW https://github.com/pgspider/sqlite_fdw

Yes.

I linked to it because it touched on BGP optimisation techniques, and illustrates the dangers of automatically passing on each update.

Is there a better link to a public postmortem that illustrates these problems in a datacenter context? When we were operating the Facebook Moonshot project we found that even internally in controlled datacenters, at a scale less than Amazon's, there is plenty of diversity

This article challenges partisans in the BGP vs OSPF debate but it's deeper than just pointing out that the wire protocol is quite superficial. Obviously, at scale you have to control OSPF to avoid update floods. But BGP also needs to be controlled so bad entries aren't trusted and propagated; for example, https://blog.cloudflare.com/how-verizon-and-a-bgp-optimizer-...

The deeper message in the article is linked https://elegantnetwork.github.io/posts/Network-Validation-wi... and it's about simulating infrastructure. This is so important for large scale systems as complex aggregate behaviour easily emerges from myopic error handling strategies that only take into account the local view from each actor. This is true for any large scale system not just networks!

The wire protocol is the surface; what information is propagated is essential and what lies underneath. Simulation illuminates these depths. Whiteboard discussions are great for debugging and communicating specific cases - for example, aggregate responses to retry backoff rules that are locally sensible can cascade into catastrophes if many actors follow the same process. However, it's hard as human beings to find these cases without the help of tools. Tools are so important!

From the outside, big organisations have massive resources and access to experts. But inside there is intense competition for those resources and people. The people are expensive and there are many opportunities so you need to justify spending their time, especially in the FAANG impact ecosystems. Despite this pressure people may misallocate their time and it can look weird if you don't hear the justifications but they are really trying hard to spend time on the right things, and they do feel constrained - there are so many rapidly growing areas that need investment, there's actually always a shortage if you think about what could be done.

Using a library versus building internally is a tough trade-off and can have long term consequences. I think everybody is comfortable with the downsides of building. But we sometimes forget the costs of dependencies. My framework is that introducing a new concept whether written internally or through a dependency has a cost. Each class hierarchy, external tool, build system, etc. has a cost and we must weigh this cost against the benefits. Dependency sprawl is expensive especially in languages like Python or JS where libraries break backward compatibility on a frequent basis. I like that Rachel is bringing up the longterm cost of dependencies, they can often be more expensive than just writing a simple function or HTTP call without supporting libraries. Building may be cheaper.

Some people just hate managing systems they can't fix. Other people enjoy the challenge of reverse engineering complex systems and squeezing the best out of them. It is only reasonable to ask for the job expectations to be explained upfront. Rachel's post about onboarding https://rachelbythebay.com/w/2020/05/22/boarded/ goes into this idea in more depth. There's so many differences even in Silicon Valley. Companies should share more honestly what the day to day will be and their appetite for building. This would restrict recruiting funnels upfront but make people happier.

The problem is that inside a company there are many perspectives. The representatives of the build faction may achieve a temporary ascendancy and will obviously try to hire then and paint a rosy picture, but then the buy faction will see the hires and become highly incentivised to point out the lack of immediate results and trammel scope, causing the conflict Rachel is writing about.

The example of Ulysses S. Grant being the best student in his class for painting makes it clear that acquisition of these so called expiring skills is associated with long term success. The list of non-expiring skills has good intentions but the entries are weak and unjustified. It's odd that the article keeps emphasizing getting along with people so many times as a non-expiring skills when President Grant was famous for being strong-willed. It might be inferred from his career that conflict even with authority is actually a valuable skill, though you could argue it goes with the many bad behaviors he exhibited which turned out to be not a disqualification for highest office.

To be successful in a competitive field like business or the military you gain advantage by improving your mechanics. These skills, like wielding a sword, drawing accurately as in the article or in software using a particular programming language, allow you to succeed at specific tasks. This causes you to be respected by other practitioners, which gives you practice in the next level of skills for larger scale collaborations.

As tools change, you have to keep up. Leadership skills like motivating people, facilitating decisions, and persisting through adversity are generalisable across situations. However if you are unfamiliar with the mechanics you will trust the wrong people, not know which questions to push on, and persist in expensive mistakes. You can't skip these expiring skills. They are an essential foundation.

The syntax follows https://github.com/krisajenkins/yesql/ which works with Clojure. So then the DB queries can be defined in a language agnostic way, so you don't need a gateway API to access the DB :)

There are many opportunities to grow this, e.g. transforming the SQL files into stored procedures, and being able to lint and check against a DB schema definition. However, I'm not sure that the indirection step of hiding the SQL away from the code actually makes sense in the long run. A tight coupling allows filters and other optimisations to be easily added to the end of a query, and saves one lookup that is not yet supported in your editor.

The plan described there is to use a signal (SIGSYS) to jump back into the process. This involves multiple kernel/userpace switches as most likely you need some Linux syscalls to emulate the MS Windows one.

I like the idea of using eBPF to run an emulation layer. That has potential to be more efficient and doesn't require the running process to cooperate so closely with the syscall emulation. Then this would probably lead to extensions to eBPF and its tooling which should be useful for other things!

The article explains that the primary benefit of user-mode green-thread fibers is not switching speed, but that they are much cheaper than OS threads, so you can have many more of them. The costs are paid in memory usage and also the operating system also has to do considerable bookkeeping for threads.

However, Netty has offered strong support for callback style IO under the JVM for a long time. This effectively allows the same efficiencies. Of course it is also possible to do without Netty. Therefore the real advantage of Loom user-mode threads and co-routines is syntactical programming convenience. That's the real innovation in Loom!

They made a related project - http://herbgrind.ucsd.edu/ - a tool that can automatically find floating point issues in real programs. You can also annotate the region you're interested in with herbgrind on/off controls.

Unfortunately - once you know you have a floating point problem, tracking it down and fixing it may be easy. Not realising it, you won't use these tools and the program will do the wrong thing :)

Real traffic exposes wrong assumptions in code, which cannot be caught with unit or integration testing. Awesome to automate this kind of testing. I encourage people to invest heavily in setting up artificial environments to replay historical data.

One benefit of the artificial environment is efficiency (can cover e.g. a week of historical data); this also requires mocking out time with simulation/replay time. The integration with data-platforms makes a big difference as it means data-scientists can help set up the right test scenarios.

Perl was the #3 language on the Tiobe ranking in 2005. It had massive codebases in banks and other industries. In 2019 it was ranked #19 and the incompatible replacement, Perl6, is #93. Breaking backwards compatibility is hugely costly. It creates immense work as people have to validate that new versions of previously working code meet the implicit, accumulated requirements.

Old code will always seem old fashioned. It will not address modern concerns. However, the option to keep using it is very valuable!

Peter Yang's book Principles of Product Management covers this. The roadmap is a set of agreed plans and priorities - the word map is not super helpful here.

As remarked in other comments - once the objectives are agreed, the product roadmap can often crowdsource itself from other functions - engineering, design, etc.

Amazon is offering 24TB RAM high memory machines on AWS. The Stackoverflow setup is far away from Big Iron :)

In terms of opportunities to add complexity: I'm surprised there is no mention of a data-platform or any model training for ranking, etc.

To reduce components, it might be possible to combine ElasticSearch and MSSQL into just PostgreSQL which has awesome text indices. As MSSQL performs very well and presents a somewhat esoteric SQL dialect this could be an expensive project.

This code creates that error by trying to wait in two threads in two loops. If the future is finished, it is fine to wait for it.

Having multiple loops waiting for each other is pretty complex to debug.

  import asyncio
  import threading

  future = None

  def p(msg):
    print(f"{msg}: {threading.get_ident()}, event loop: {id(asyncio.get_running_loop())}")


  async def one_time_setup():
    p('one_time_setup start')
    await asyncio.sleep(1)
    p('one_time_setup done')

  async def maybe_initialize():
    global future
    if not future:
        future = asyncio.create_task(one_time_setup())
        p('waiting start')
        await future
        p('waiting done')

  def worker():
    new_loop = asyncio.new_event_loop()
    asyncio.set_event_loop(new_loop)
    new_loop.run_until_complete(maybe_initialize())

  number_of_threads = 2
  for _ in range(number_of_threads):
    threading.Thread(target=worker).start()

this prints

  RuntimeError: Task <Task pending coro=<maybe_initialize() running at test.py:20> cb=[_run_until_complete_cb() at /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py:158]> got Future <Task pending coro=<one_time_setup() running at test.py:11> cb=[<TaskWakeupMethWrapper object at 0x10cafdeb8>()]> attached to a different loop

exactly -

The problem is that there are three logical states that need three different actions

1. if nothing done yet: start async computation

2. if async computation in progress: wait for it

3. return result as computaton done

Atomicity in the sense that no other Python code can run during a setdefault, is not relevant.

Waiting for something that is running in another loop actually isn't easy either; I'm not sure that Python really allows it - won't you get an error:

   The future belongs to a different loop than the one specified as the loop argument

The databases I implemented experimented with various ways to compile queries. Turns out that the JVM can run quite fast. I feel like LLVM (Julia) is likely to be able to be better for throughput and definitely better for predictability of performance.

If you know layouts and sizes, then your generated code can run really fast - using SIMD and not checking bounds is a win.

Hugepages would greatly reduce pagetable bookkeeping, but obviously may magnify writes. Wish I could have tried that!

mmap'd databases are really quick to implement. I implemented both row and column orientated databases. The traders and quants loved it - and adoption took off after we built a web interface that let you see a whole day and also zoom into exact trades with 100ms load times for even the most heavily traded symbols.

The benefits of mmaping and in general POSIX filesystem atomic properties are quick implementation, where you don't have to worry about buffer management. The filesystem and disk block remapping layer (in SSD or even HDDs now) are radically more efficient when data are given to them in contiguous large chunks. This is difficult to control with mmap where the OS may write out pages at its whim. However, even using advanced Linux system calls like mremap and fallocate, which try to improve the complexity of changing mappings and layout in the filesystem, eventually this lack of control over buffers will bite you.

And then when you look at it, the kernel (with help from the processor TLB) has to maintain complex data-structures to represent the mappings and their dirty/clean states. Accessing memory is not O(1) even when it is in RAM. Making something better tuned to a database than the kernel page management is a significant hurdle but that's where there are opportunities.

There are many good points here. Companies depend on in-person proximity for automatically communicating. The article identifies a few cases where remote structures fail, like in responding quickly to new circumstances but it's a more general problem.

Ideally, groups manage to articulate priorities in a planning process and share what people are working on. However, these processes sometimes do not exist, and when they do they rarely identify all the work. People are regularly drawn into unplanned work, e.g. helping other teams. Also, to excel as an knowledge worker (e.g. software engineer) you need to have a few experimental projects, too risky to fit into the planning process and which should not be discussed there. Getting help and mentorship on these can greatly accelerate career growth, and that's often what people do late at night in the office. This is definitely not inclusive, but is how people signal that they are going above and beyond.

Remote work exposes the gaps between official processes and the informal ones that spring up to enable human collaboration. Making these explicit would be valuable for fairness, to help people without experience understand what is really going on.

Doing that is very hard. If these informal power dynamics were documented, indubitably they would be improved; they are very often unfair. Another way of looking at it, is that the informal processes typically fill in gaps in poorly designed formal processes :) It would take extraordinary organisational maturity to be comfortable sharing this dirty laundry.

In summary, companies that go more remote will find ways to fill the gaps that are currently patched over by in person proximity, and this is for the best as it means tackling tough cultural painpoints

The value of comments and references depends on the software context. If inexperienced people are expected to be editing or reading the code they're more valuable. However, they are expensive to keep in sync with changes to the implementation and the value has to balanced against this cost.

Java and C# did not replace C. They're typically used for different tasks.

Objectively speaking, do we have evidence these enterprise OOP practices lead to better software?

It's hard to do objective research. Some studies try A/B tests on student volunteers. But then this setting is clearly different to professional teams working on a project for a long time.

The value of new methodologies, languages, and techniques is partly that the enthusiastic proponents of them are given a chance to prove out that there is value, and so become motivated to go the extra distance to achieve the project specific outcome.

This value is destroyed if people are forced to use the technique, instead of championing its introduction. So measurement is made even harder!

Language X is at least C++ (as linked), Java https://developers.google.com/protocol-buffers/docs/referenc... , and Python https://github.com/protocolbuffers/protobuf/blob/master/pyth...

I think the Protobuf spec focuses on the binary serialization - the text format and JSON representations are not related to that at all, of course.

Great points!

FAANG companies like to manage people without giving negative feedback. This can be very confusing if you are used to explicit feedback and come from a more direct culture. "Listening to your manager" can be difficult if you are not used to decoding the issues.

Coasting is often possible. But this is not a good strategy - you are throwing away a huge opportunity to excel, grow and take on more challenges and scale. If you are not having a great time with growth on your current team, it's not advantageous to do the minimum. Instead, look for more exciting projects!

Why do FAANG companies allow coasting? It means people can feel safe, find passions and ideally excel and uncover huge value in these growing industries. Rather than a visible stick, there are (invisible) carrots. Take advantage of these!

As noted, eventually, lack of progression will count against you. You will be overlooked when big chances come up and fail to build relationships that will aid you in your career. It's hard to hire people who find these tasks easy - showing your capabilities will open doors.

Protobuf actually has a canonical text format. It's easy to produce in C++ or Java https://developers.google.com/protocol-buffers/docs/referenc...

The format has much less extra syntactical noise than JSON.

For example,

    name: "vii"  # comments allowed!
    id: 23923373

Pretty nifty as it allows readable configuration files with structured data.