vaughan

We should break out of the ide and visualize code more. I think then we would see the tangled mess we are creating. And then if we started talking about code that was cleanly visualized, we would truly have understandable code.

We need two-way sync for visualizations of all data structures and data flows. And then we need to see the actual data values inline as it flows through the system as we read the code.

Wallaby.js is a great leader in this space. I found it too hard to setup and too slow - but I’m convinced this applied to the entire stack is the future.

We draw diagrams on the whiteboard to explain every system but they are completely absent when we actually code.

When we ask ourselves: "how should we structure our code?", the primary goal should be: so that it's easily visualizable and debuggable. Our guiding principles at the moment are all over the place. For a long time it was: make code testable. Then we had things like: eliminate side-effects, immutability, one-way data flow, static type-checkable, etc.

The problem is that by ignoring visualizability, when you do come to visualize it (which everyone inevitably needs to when reading code and building a mental model), it's this huge tangled mess. People forget that the purpose of structuring code is to make it easy for HUMANS to understand. The computer only understands assembly at the end of the day. So anything beyond that is suppose to be for our benefit.

I think its a problem with the way we write software. I think all code we write should be visualizable at different levels of granularity, and should be able to show data flowing through it inline. We have all these principles about writing good software such as it should be testable, performant, self-documenting...but I would argue the most important thing is that it should be visualizable! And easily debuggable.

---

I would suggest using a visual debugger and an IDE with static call hierarchy features (sometimes called find usages or find all references).

You should think about how you would build it from scratch. Then fit the existing API to your own mental model of how you would build it. Then it will be easier to understand why people did things a certain way. Many times I will see someone's code and wonder why they came up with such a complicated abstraction layer, but then when you build it yourself you see how it comes very naturally. The problem is for most projects you are looking at the end result of someone else's journey of building from scratch and refactoring along the way as needed.

> A better approach is to accept that there are irreconcilable differences between local and distributed computing, and to be conscious of those differences at all stages of the design and implementation of distributed applications. Rather than trying to merge local and remote objects, engineers need to be constantly reminded of the differences between the two, and know when it is appropriate to use each kind of object.

Aka. Transparent remoting.

I agree with sentiment of the paper in that it can never be fully transparent. When I say "it should just be a process of identifying hot function calls..." this is admittedly a little contrived - there would need to be a more thorough analysis.

The main point seems to be that partial failure is too difficult to handle without explicitly declaring some objects as "remote".

> Merging the models by making local computing follow the model of distributed computing would require major changes in implementation languages (or in how those languages are used) and make local computing far more complex than is otherwise necessary

Given that the paper is from 1994, I'm wondering if there have been language changes that make this more achievable.

What comes to mind is the event loop. The fact that in Node.js, almost every function call is now an asynchronous promise due to the non-blocking paradigm coupled with the convenience of async/await syntax. These event loop runtimes provide a universal way to hook into asynchronous calls.

I would imagine that in 1994, envisioning the code added to the languages of the day needed to handle timeout and asynchronous invocations would have felt overwhelming.

Also thinking about ORMs and databases. It seems TOPLink for SmallTalk was the first, released in 1995 after this paper was. ORMs seem kind of like "transparent remoting", as they usually can run against an in-memory database or db on a local system, or over an TCP connection, which then needs to deal with latency and failure. And for any app talking to a DB with an ORM, means that the failure cases of the ORM must prevade throughout the stack.

The main thing is that regardless of scaling, the app should always be able to run/debug/test locally in a monolithic thing.

Once people scale they seem to abandon the need to debug locally at their peril.

Scaling should just be a process of identifying hot function calls and when a flag is set, to execute a call as a network rpc instead.

Think: Reddit, HN, Twitter, Discord, TikTok, Insta, Signal/Whatsapp/Telegram/Messenger.

I feel like Reddit is a superset of most though although they suck at chat. Discord is great at chat, but their forum functionality is weak.

All of HN could be implemented as a single sub-reddit for example. I would much rather use HN on the Reddit UI too. And Twitter also.

The most important thing that trumps all is always the community.

I've often thought a fun experiment would be to create a simple clone of the UI of Reddit, HN, Twitter, Discord, and then show each of those feeds inside the UI of each other.

Blocking.

Instead of bans, Twitter could just add people to their own block list, and then tell new and existing users that they can opt-in to a moderated Twitter experience. They could have different block lists for different moderation reasons. E.g. Bots, spam, vs terms violations.

An even better solution is a delegative blocking system for accounts and keywords. It's basically crowd-sourcing account blocking, where you are saying: "I want to block all accounts and keywords that this person blocks". Then you can have an allowlist to override some blocks.

Don't like conspiracy theorists? I am certain there are many people willing to tag all the conspiracy theorists they find.

The best part would be having a toggle that can show you all the content that is blocked, and why it was blocked (i.e. You delegated all your blocking to account X, and they delegated their blocking to account Y, and account Y placed this account on their block list named Z).

You could also have "smart block lists" where you could use queries to block people. E.g. "Block all followers of X". An issue you might have though is to differentiate between those that follow in support of someone, or follow simply to know what they said and challenge it.

Twitter used to have block lists but they got rid of the feature. See: https://help.twitter.com/en/using-twitter/advanced-twitter-b...

Ideally I would like this for all internet content.

So I agree with Jack when he says:

> The biggest mistake I made was continuing to invest in building tools for us to manage the public conversation, versus building tools for the people using Twitter to easily manage it for themselves

Just as long as its always opt-in for everyone individually.

When a source table changes, I want it to automatically and efficiently update anything that would change. I think pretty much every system would prefer real-time stuff like this.

If you are doing this with SQL you will start looking into "Incremental View Maintenance" which is quite complex and still quite heavy.

Then you realize that if you take control instead of handing it off to the planner, you could code your joins and transformations by hand, cache intermediary steps as needed, and have a query that is more efficient than SQL.

But for this I would argue you need a better way to visualize your data flow and the dependency graph, because people can easily write slow imperative stuff.

I think this is where people have got it wrong in the past.

This tool is an example of something in the right direction: https://www.ilograph.com/

It's just completely disconnected from code.

I think the key is to not make everything visual. You want to be able to view your system as you would draw it on a whiteboard. I believe it could be done using a restricted syntax of a proglang.

> Teams I've been close to lately have taken the python pill...

Most people have a desired result and want it as fast as possible.

When imperatively using map, groupBy, filter, you can usually get the immediate result you want very quickly. This is the appeal of Pandas, Python notebooks, etc.

But when someone asks: I want to re-compute on change, which pretty much applies to every task...their quick solution is not optimal.

And then if the schema changes, and you're doing all this intermediary stuff...then refactoring is next to impossible.

Any SQL query could also be written in an imperative prog lang and its probably easier to understand and tweak and also trace. An SQL relation does not maintain any info about its source, which then makes incremental updates difficult, and also refactoring.

You can basically ask the question: for every piece of data you see in your company, can you easily see the "data lineage" from its source and all the transformations it was involved in, and how clean and efficient is this pipeline when receiving new data. Then a step further is, how easy is it to change something.

I'm convinced this entire space should be visual. We always visualize data pipelines in our minds and on whiteboards...well, visually. Two-way code-diagram syncing is needed to allow best of both worlds. We should serialize to yaml from imperative code, but allow manipulating the diagram to modify the imperative code.

Ideally what you want is also to track the dependencies of every atom of data in your org, and then have something cache and incrementally compute updates.

The biggest thorn in this vision is SQL and the relational model, and I don't think a lot of people realize it.

It favors representing data such that the query planner can optimize query execution...rather than being able to track data dependencies and visualize data flow. It wasn't designed in mind with complex ETL pipelines and many external data sources of today's world.

    person->eat(bread)

100%. I think this is the next wave of development. Back to the future.

> Not monkey about with YAML and JSON.

The key difference this time needs to be two-way sync between code and visual.

Having a serialized format (YAML/JSON) for wiring at some level is important though, but it should be easily modifiable by humans.

In the last wave, we left this two-way syncing behind. An example of this is XCode's NIB files and Interface Builder. NIB weren't designed for humans to modify so everything had to be done through IB which made certain things a pain, and created a lot of VCS churn.

I've been thinking about whether we can achieve a two-way syncing (text <-> diagram) visual programming interface by interacting with the AST of a (subset of) an imperative language and using data flow tracing (via code instrumentation).

I wonder what the minimal data structures needed are to represent most of configuration programming. Such as state machines, streams, graphs (dependency tracking).

Otherwise they are using libgit2.

It's interesting to think about the difference between an API as a library vs CLI / REPL. Often when I am building a library, for testing I usually would like almost every function to be runnable as a CLI.

Anytime someone is doing some piping (joining commands) or shell-scripting, it usually could also be its own script with a CLI. Many applications could also just be shell scripts piping stuff together which I think is actually what the unix people envisaged. Starts making you ask: why are some things CLIs and not everything. Lots to think about...

For source control, IntelliJ does actually print all the git commands it is running which is nice.

CLIs all have the same consistent user interface. Positional arguments, flags, and text output. On unix-like envs they also have pipes. This is why people like them. This is what makes people productive in them.

Sure, modern GUIs have a few shared UX paradigms, but largely they are all different.

I wonder how the unix philosophy applies to GUIs? Or what the early developers thought about it. How would piping work in a GUI.

A compromise is that GUIs should print the commands that can be run to get the output they are rendering and to perform the actions they are doing. Like a little command palette at the bottom of the window.

Then the user can always break out into terminal.

Then in the output, I have a bunch of make tasks and child make files running. I care about the output for each makefile as it runs, but then would prefer it to collapse when its finished. Otherwise its too difficult to see each step of the build process. A terminal cannot do this. XCode does it kind of well.

At the end of build too, when it reports errors, I'd like to jump up to the errors.

Almost every command output I would prefer to view as a GUI table or as a webpage.

The problem is that then, instead of just printing output, now I am dealing with a huge mess of frontend frameworks and build toolchains.

Everything is relatively simple to understand when visualized like it would be on a lecture slide. But reading about it in manpages and blog articles is overwhelming.

Terminal is good because its unconstrained. But most of the time I would prefer a GUI interface.

We just need to blend the two approaches better and always have a terminal fallback.

I agree with your point but the analogy is bad. I would definitely buy a car based on factory. Reliability, order time, parts availability, etc.

Imagine being able to easily step through the entire execution from user to kernel space. And also step backwards. And also record replay.

And then make instant changes during dev to any layer of the stack.

Think about how much time is spent across so many different tools trying to figure out perf bottlenecks.

Then think about if you wanted to patch something or write a driver, how difficult it is.

Then you would also want the entire system visualizable.

Imagine if the entire data flow and data structure of the system were automatically visualizable by instrumenting the source code. Why do we draw diagrams on whiteboards if they are not the best way to understand code.

Do this first and the rest becomes easy.