shalabhc

avro supports field renames though.

3. on second thought i believe you'd only have to deploy when you choose. the next build will force you to provide values (or opt into the default). so forcing inspection of construction sites seems good.

1. Dense json

Interesting idea. You can also just keep the compact binary if you just tag each payload with a schema id (see Avro). This also allows a generic reader to decode any binary format by reading the schema and then interpreting the binary payload, which is really useful. A secondary benefit is you never ever misinterpret a payload. I have seen bugs with protobufs misinterpreted since there is no connection handshake and interpretation is akin to 'cast'.

2. Compatibility checks

+100 there's not reason to allow breaking changes by default

3. Adding fields to a type: should you have to update all call sites?

I'm not so sure this is the right default. If I add a field to a core type used by 10 services, this requires rebuilding and deploying all of them.

4. enum looks great. what about backcompat when adding new enum fields? or sometimes when you need to 'upgrade' an atomic to an enum?

It goes into not only different isolation levels, but also some ambiguity in the traditional ACID definition.

I believe a 2nd edition is imminent.

What could be useful here is if postgres provided a way to determine the latest frozen uuid. This could be a few ms behind the last committed uuid but should guarantee that no new rows will land before the frozen uuid. Then we can use a single cursor track previously seen.

https://dagster.io/blog/fast-deploys-with-pex-and-docker

Computation is a physical process and any model we use to build or describe this process is imposed by us. Whether this model should include the concept of data (and their counterparts "functions") is really the question here. While I don't think the data/function concept is essential to modeling computation, I also have a hard time diverging too far from these ideas because that is all I have seen for decades. I believe Kay is challenging us to explore the space of other concepts that can model computation.

* It can be used as schema-less

* allows attaching metadata tags to values (which can serve as type hints[1]), and

* encodes blobs efficiently

I have not used it, but in the space of flexible formats it appears to have other interesting properties. For instance it can encode a symbol table making symbols really compact in the rest of the message. Symbol tables can be shared out of band.

[1] https://amazon-ion.github.io/ion-docs/docs/spec.html#annot

* https://en.wikipedia.org/wiki/A_Symbolic_Analysis_of_Relay_a...

Do you handle the case where the actual objects don't overlap but result of an aggregate query is still affected? For instance a `count(*) where ..` query is affected by an insert.

The other side of something like Spanner is the quorum-based latency is often optimized by adding another cache on top, which instantly defeats the original consistency guarantees. The consistency of (spanner+my_cache) is not the same as the consistency of spanner. So if we're back to app level consistency guarantees anyway, turns out the "managed" solution is only partial.

Ideally the managed db systems would have flexible consistency, allowing me to configure not just which object sets need consistency but also letting me configure caches with lag tolerance. This would let me choose trade-offs without having to implement consistent caching and other optimization tricks on top of globally consistent/serializable databases.

You may find other interesting articles linked from here:https://en.wikipedia.org/wiki/Jamie_Zawinski, eg https://www.jwz.org/gruntle/nomo.html

K8s may have some more controls for the "incremental deployment" case but I'm less confident about the isolation between pods to run user provided code.

There are two sides to the problem: the build and the deploy.

> it feels like just having a build machine with storage would immediately solve the problem, there would be no remote pulling. Is that not a solution?

Indeed, ephemeral builders are an issue and this would be an alternative solution. However doing this for an arbitrary numbers of users who want on-demand deploys for a variety of projects is non trivial. Most CI environments give you a "clean environment" at startup.

Even if the build takes 0 seconds, the other part is re-running this docker image (where 100s of MB is unchanged and say the final 1MB layer changed). You need to ship all these layers to a provisioned container and boot it up. Even this is possible to optimize by heavy caching and optimizing the container service - but that's an alternative solution with different trade-offs. With pex, we can update any docker image in-place so it is container service agnostic. It also works with our current service (Fargate) which has famously slow startup times.

Irrespective of the CI, I believe all cached Docker layers will need to be downloaded onto the build machine before it can be rebuilt.

Still, I believe it is possible to build and deploy faster even with a "docker image only" design and it's something we are still looking at. The question is what is the lower bound here - would be hard to beat "sync a file to a warm container and run it". Pex gives us a pretty good lower bound that is also container platform agnostic.

Firecracker is definitely very interesting. Would require more ops work for us to run bare metal EC2 (we currently use Fargate). IIUC reusing pre-existing environments would require us to share ext4 filesystems across the VMs. Not sure if antistrano helps here but will look into it.

I agree it would be fantastic to have sub second deploys! Doing this for user specified Python environments is challenging in different ways than doing it for a JS SDK like Workers. Note that just provisioning a GitHub runner takes about 10s, before any deploy code even starts up. In theory we could rsync the code directly to the code server and reload it. But any of these options require bigger architectural changes.

Also you can already have a local environment setup for faster iteration and use this for the shared environments with other developers, where the speedup is still great to have.

You can already do local development with dagster, if you set up a local environment. Some users may not want to set up a local environment with all dependencies, secrets and so on, so they can use the remote environment option. Remote environments are also easy to share with other developers and can be used in addition to local environments.

It's a pelican based static site. More details of the tech used is here: https://shalabh.com/pages/site-tech.html

1. define a view representing the object fields you want to cache 2. define eviction and properties of the cache 3. use queries to query either the cache or the db

Specifically, do all of the above using edgeql and not have to write the cache consistency or serialization/deserialization logic.

I think EdgeQL (and the data model) has potential for being a lot more than accessing the database. For example, you could have a distributed system use this data model across various typically replicated data stores - include all caches and "materialized views" of various kinds. Extend EdgeQL to define various caches and computed values from the core schema. Then you could use it to represent querying the same data but with different freshness. Now this becomes a compelling idea. Essentially it gives you a well defined way to declare and manage various computed values, laggy caches without having to manually implement all that coordination.

Yes exactly. Except it would be much faster, more precise and with more coverage than what we get with mental execution. It still wont give you total coverage, but it would be able to show where it doesn't have coverage.

> In a way, it's just a different interface for doing the same thing you might be doing with unit tests - except it would be transient, and free to exercise any internal code in your program, not just poke at public module interfaces.

Yes. One additional aspect would be the ability to run with partial information. Eg. if a function needs three arguments, you need all three to run a test. However in abstract interpretation, you could just specify one argument and still trace the execution to some degree.

I agree. Here gtoolkit is a recent example that's exploring the building of custom visualizations very easily. It's a compelling environment.

> Secondly, moving to a new programming language (or even a new development environment) is not something that most of us have the freedom to do in our daily work, and overcoming that inertia is a real challenge.

Couldn't agree more. But over time, things could change. Python was a small language a couple of decades ago.

> rather than looking for a panacea or a magic bullet, we can just try to use the tools that we have today to get as close as we can to the vision that so many of us share

Yes this is one way to move forward. The main issue I've seen is supporting 100% of Python becomes too hard since this idea is being retorfitted and the language/environment hasn't been designed with this from the start (consider getattr/setattr and IO). So certainly a large subset of Python (or another dynamic language) withing a custom environment could work as a starting point.

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.575....

Excerpts:

"DSEE is implemented as one program, with instances running at various nodes in the network."

On the history manager:

"DSEE can create a shell in which all programs executed in that shell window transparently read the exact version of an element requested in the user's configuration thread. The History Manager, Configuration Manager, and extensible streams mechanism (described above) work together in this way to provide a "time machine" that can place a user back in a environment that corresponds to a previous release. In this environment, users can print the version of a file used for a prior release, and can display a readonly copy of it. In addition, the compilers can use the "include" files as they were, and the source llne debugger can use old binaries and old sources during debug sessions. All of this is done without making copies of any of the elements."

On the configuration manager (sounds like system wide built artifact caching):

"The CM maintains a derived object pool which holds several version of each object that was produced as the result of building a component named In the system model (e.g., binaries). Each derived object in the pool is associated with the ECT used to build it. When asked to build, the CM determines a "desired" BCT by binding the system model to the versions requested by the user's current CT. The CM then looks in the derived object pool to see If there ls a BCT that exactly matches the one desired. If a match is found, the derived object associated with that BCT is used. Otherwise, the component is rebuilt In accordance with the desired BCT, and the new derived object and BCT are written to the pool. In all cases, the user is given exactly what he asked for."

But if we actually look at what was around, both in theory and practice, CVS was a giant leap backwards.

Examples:

1. Smaltalk ENVY (1990s, I think?): Automatic class/method level versioning on save. Programmable, introspectable history to easily build CI type stuff. See user comments here: https://news.ycombinator.com/item?id=15206339.

> you could easily introspect every change to code and by combining introspection of classes and methods quickly determine which changes were happening where. We built a test framework that could list the changes since the last run, then compute dependencies and run an appropriate selection of test cases to check. This cut down test time by 90%

2. DOMAIN Software Engineering Environment (1980s): a distributed source control and config system where the provenance of built artifacts to source files was maintained. More than that:

> DSEE can create a shell in which all programs executed in that shell window transparently read the exact version of an element requested in the user's configuration thread. The History Manager, Configuration Manager, and extensible streams mechanism (described above) work together in this way to provide a "time machine" that can place a user back in a environment that corresponds to a previous release. In this environment, users can print the version of a file used for a prior release, and can display a readonly copy of it. In addition, the compilers can use the "include" files as they were, and the source line debugger can use old binaries and old sources during debug sessions. All of this is done without making copies of any of the elements. (from Computer-Aided Software Engineering in a Distributed Workstation Environment, 1984, http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.575...)

Note again, the above is from 1984.

3. PIE reports (1981): Describes a model of "contexts" and "layers" (roughly analogous to branches and revisions) for nodes (not files) that version methods, classes, class categories and configurations. On merging work from multiple authors:

>Merging two designs is accomplished by creating a new layer into which are placed the desired values for attributes as selected from two or more competing contexts.

(from An Experimental Description Based Programming Environment By Ira Goldstein and Daniel Bobrow, 1981, http://esug.org/data/HistoricalDocuments/PIE/PIE%20four%20re...)

Modules (which are just top level classes and contain nested classes) have no import statement and no hard linked external dependencies. When you instantiate the module (~class) you pass in dependencies it needs.