The Humane Representation of Thought [video] | Hacker News Reader

The Humane Representation of Thought [video]

vimeo.com

196 pointsjashkenas32 comments

The interesting part is that a lot of what he is talking about is already possible.

- You can use augmented reality or VR with something like Oculus to display things in 3D space.

- You can use Motion Capture technologies, plus maybe some convenient remote-control/keyboard-like devices to put a human in that virtual space and manipulate things.

Only 2 technologies(Oculus + Mocap) would be enough to accomplish something like this: http://youtu.be/VzFpg271sm8

Give a bunch of people oculus rifts and put them in the same virtual reality - and we have the dynamic stage he is talking about.

They don't even have to be in the same room of course, we have 3D realtime MMORPGs and second-life that already solved that problem!! You could attend a virtual talk by sitting in your apartment with oculus rift on your head.

And of course "downloading a room" instead of a book is already very doable. We already have computer games and engines. That room is like a video game level. Downloading it is not a problem, to explore it all you need is oculus and a simple game controller or a keyboard/mouse.

And in terms of creating such content, the closest thing I can think of is 3D editors, like Maya or Houdini, especially houdini(http://www.sidefx.com), which is the brilliant combination of computer graphics and programming, that allows you to model and program dynamic things visually.

I've also been thinking about the concrete representation of symbols and the negative "inhumane" impact of screens, but my conclusion was much different than Victor's: screens take us away from the fullest experience of world, and so their use should be minimized. This is basically Victor's conclusion minus the possibility of what I might call "somatic programs" at scale.

That screens are problematic asserts a moral imperative for programmers to avoid writing applications that are designed to be experiential cul-de-sacs. This, in turn, goes very much against the grain of most technologists and the people who fund them, who are all looking to make the most potent screens they can to pull people out of real life as much as possible. This makes great economic sense, but it is unethical.

Most of us who are so fundamentally tied to a screen for so many years at a time would do well to systematically disengage and interact with the real-world as much as possible. What I wouldn't give at this point in my life to put on a hard-hat and help build a bridge!

I'd push back on the dog in a cage analogy. I think it could be argued that we've narrowed our communication mediums for purpose of effectiveness and efficiency. Also affordable ubiquity. I would argue it's more like limiting a fish to water rather than a dog to a cage. Not to say it can't be improved in light of today's tools though.

Great talk overall, a lot of interesting insights!

The analogy was an example of something that is inhumane for analogous reasons that restricted media is inhumane. It doesn't allow for full dog-life or full intellectual-life, some critical and important experiences and behaviors are excluded.

Changing the analogy to a fish in a tank suggests an assertion that these other intellectual modes are not important for human experience and wouldn't make communication more effective and efficient.

To tweak the analogy further: Otters are very good at swimming, but their habitats should include land.

Putting a dog in a cage is inhumane because one actor has taken away liberties that the other actor is known to have "by default." We know that dogs are naturally able and inclined to run around and play, and we have intentionally restricted that.

Personally I don't think his usage of the term "[in]humane" to describe human media is appropriate. Who is responsible for this inhumanity? Our entire race is collectively treating itself inhumanely by not inventing entirely new forms of media and expression? That seems like a stretch. Are we inhumane to dogs because we haven't yet helped them discover their fullest and richest capabilities as a species, whatever those may be? I love Bret and his vision and talks, but I feel like this particular usage is heavy-handed shame rhetoric (I get that he is trying to light a fire under people, but it still seems forced).

some critical and important experiences and behaviors are excluded.

What defines experiences as critical and important, such that to exclude them would be inhumane? Is it inhumane that I haven't traveled on a spaceship to Mars? That seems like a pretty important experience. But of course that's ridiculous, because you can't include the infinite landscape of experiences that might be possible. Personally I don't think either analogy (dog or fish) is particularly apt, but I think the fish one is better because a) our generally known and accepted capabilities are not being forcibly restricted by another agent, and b) the stuff Bret is talking about is entirely new and not possible within our natural environment (i.e. there are many natural and accessible ways to write symbols on a surface, but nothing like that for arbitrary-dynamic-physical-hologram-things).

I don't know why anyone has to be 'responsible' for an inhumanity to exist. One actor taking the liberties of another actor is not required.

A) Our capabilities are being restricted by the circumstances we find ourselves in. A dog happens to be in a cage built hundreds of years ago. There must be some words to describe this situation, I think inhumane applies.

B) Now we can start opening the cage.

Edit: our natural environment is inhumane and awful. Your commentary seems to heavily favor the status quo in a way I can't relate to.

The additional inhumanity of media is from 2 things: the fact that information processing has been mostly limited, and that information processing has become really important lately, so a lot of people are spending a lot of their time in that limited environment.

Call me obtuse and change my view please:

I'm actually quite interested in this spatial learning thing. I had an aboriginal acquaintance who explained to me how his culture would attach meaning to places and even physical orientation (North South East West etc). However, it has been so far very very difficult for me to wrap my head around that. Visual notation imo, has been the most efficient way of communicating between humans. It also helps a lot in understanding.

A lot of what he says harks back to a 1980s/1990s/early 2000s style of teaching with an increasing use of "aids". Using props to teach math is a bit silly (well, asides from geometry, which is inherently spatial). I really cannot see how we can represent the learning of linear algebra in a dynamic spatial manner.

How for example, would one represent higher dimensional spaces spatially, when you really are limited to only 3 dimensions in real life? I am aware that Iron Man style projection tech may actually help in visualizing it, but again, that's a visual aspect of learning, not spatial. The most efficient way of doing so unfortunately is still in algebraic notation.

Perhaps I'm doubtful because the use of props, spaces, songs, moving your body etc has never worked for me in school. Ever. Can anyone share any studies done on the use of these cheesy 1980s-1990s educational aids (the closest analogue to all the things Brett mentioned) in aiding learning?

EDIT: If you're interested, there's been a good book that I read, in attempt to wrap my head around the concept (however to no avail) - Wisdom Sits in Places.

One of the educational techniques you're alluding to is probably TPR (total physical response), for foreign language teaching.

https://en.wikipedia.org/wiki/Total_physical_response

I had a Spanish teacher and later a Latin teacher (teaching adults) who favored TPR and would tell people to jump a specified number of times, or to touch their heads, and so on. I don't have a clear opinion about the effectiveness of TPR but I thought I'd mention the name so you could use it to search for research if you like.

There is also a century-old approach to using colors to visualize 4D geometry (I think the color represents the offset in the 4th spatial dimension), proposed by Charles H. Hinton; they're called "Hinton cubes". Martin Gardner describes them and reports that some people have found them too effective, I guess becoming mildly obsessed with them or having the learned associations interfere somehow with day-to-day visual perception.

When I learned/taught myself algebra in an experimental school in 6th grade, we had these alegrba blocks and 2x2 grid, the left/right sides representing the sides of the equation and the top/bottom sides representing postitive/negative.

There are some pretty big limitations on how far in algebra you can go with that. I do think it created a solid foundation for my mental model as I transitioned to only using symbolic equations.

i think the word learning is a misnomer here, the term learning in itself is dubious enough

look to the best line, and i'd argue one of the most important ideas to ingestively portray to a mainstream population, in 2008 cinema, also an argument could be made to increase that reach.. perhaps in the last decade, century, however hyperbolically you care to extend the reach.. from the film burn after reading(i):

    CIA Superior: What did we learn, Palmer?
    CIA Officer: I don't know, sir.
    CIA Superior: I don't fuckin' know either. I guess we learned not to do it again.
    CIA Officer: Yes, sir.
    CIA Superior: I'm fucked if I know what we did.

i think what victor is suggesting here would aide moreso in efforts to improve understanding than facilitate a traditional understanding of learning

when quantum mechanics was young there was a sudden rush of interest in quantum understanding and one of the modes of expressing improvements in that understanding was to create models(ii) sometimes completely replacing old models, if i remember victor even says bohrs name in relation to this concept

this i think is the sorts of things victor is alluding to, stead seeing it as learning think of it as new perspectives to improve understanding.. interestingly though one would have to learn the concepts, both current and avant, in order to know when a new expressive medium improves on the old

so knowledge accumulation is a part of this sort of environment, but arguable only subordinate to the true goals

the shot of the stick figure pushing on a bond between elements in a molecular model i feel is the best representation of this idea

i'd love to have a room like that:

   a central tool with supporting data surrounding it
   the central tool is imbued with evidencial data w
   when i manipulate the tool new data is expressed in different ways around me
   i apply my changes physically
   the the data maps all changing in sync

lets say i want to try to bond a new element in place of an old one, i can do it physically and watch as the object changes before me

in the picture the stick figure is applying force on that bond, perhaps the idea the figure is testing is what if i bend this bond or strain it, what then? or if i move this bond over here does this better represent our understanding of molecules? .

my issue with the talk was two fold, i thought victor tended to talk down efforts others have already made or are making to do something with similar goals but in a wholly different means, and victor also failed to mention the number of people already doing the very things he is suggesting, it came off as unnecessarily dramatic and i found those were the times when he really lost me to wandering thoughts

and also i found it interesting that victor seemed to suggest this new dynamic reality would manifest without the use of current forms of creation

things like, ~programming of the future will be different than programming of today~ which i both agree and disagree with, i think both the current and future ways to program will be in tandem and it will be more of a matter of who touches what aspects of it,

i think like hardware design and manufacture programming will become a specialised field

look at compilers, that is people really interested in low level bit moving granting people with high level interests.. in the programming sense stead some cogitive nobile sense.. access to their expertise from within the comforts of the higher level

the same i believe will be true but that gap will likely grow as it has with hardware manufacture

there will always be a need and opportunity for tinkers to challenge the manufacture to keep it fresh, like the hardware scene(iii), but realistically this shift in software will most likely have the same dramatic effect on quality of life as the hardware model has

(i) http://www.imdb.com/title/tt0887883/?ref_=ttqt_qt_tt

(ii) https://www.timetoast.com/timelines/atomic-model-timeline--3

(iii) https://www.youtube.com/watch?v=PdcKwOo7dmM

First a few nitpicks

1. Walking around a gallery and waving your arms is not enough exercise 2. Incremental progress is giving us VR through Tactical Haptics, Oculus Rift, etc. He says his thing is not VR but R or dynamic R. The distinction is unclear, sounds like marketing.

I also had the idea of talking about systems, particularly political systems through models and simulations. Having spent too much time talking about politics, economics, and computer holy wars in text, it feels like a few interactive models could have saved all that time for millions of people.

That said, simulations are widely used to communicate about financial topics, mainly spreadsheets. And the effects of using spreadsheets are pretty interesting https://medium.com/backchannel/a-spreadsheet-way-of-knowledg...

People start thinking in spreadsheets. A lot of the weird merger acquisitions since the 1980's were basically caused by them. Economists already use fancier models. And as with spreadsheets, garbage in, garbage out. Still I feel they'd be useful in casual conversation.

1. If you are doing it all day, it would almost certainly be sufficient.

2. He's talking beyond VR. He never says VR isn't a stepping stone. But he does make a good point about it being dehumanising.

That last part catches my interest. Could someone do to the agent-based model what the spreadsheet did to arithmetic, i. e. make it low-effort and scalable?

Great talk and ideas. But the idea of representing mathematical equations in the physical form is like supporting the same medium of symbols and just representing it in a different way. But fundamentally math was invented using pen and paper, so the centuries of mathematical thoughts were already slaves to the 2D medium. Maybe it would be interesting to think about, what would happen if we take away 2D surfaces and ask people to reinvent math in a medium that Bret is talking about. How would someone go about discovering, solving, proving mathematical systems without pen and paper? How would you represent 4, 5 or 20 multi-dimentional spaces in a 3D environment ? Maybe pen&paper was kind of abstracting out limitations of physical world so that mind can explore even further.

The progress in n-category theory is largely about that. Instead of just studying objects, operations and equations, like "a group is a set g, together with multiplication + certain axioms" one studies objects, relations between objects, relations between relations etc. To study this up to level two, you visualize objects as regions in 2d space, relations between objects as lines between them and relations between relations as points where lines meet. Equations are then equalities between such pictures. If you look at those equalities as a dynamic process, which starts at time t=0 with one picture and time t=1 with the other, you get a picture in three dimensional space, which you can visualize as a movie (to be mathematically precise after the choice of a morse function you get a movie). One of the easiest examples would be the zig-zag identity. See for example https://golem.ph.utexas.edu/category/2006/10/classical_vs_qu... and scroll down.

One person who has completely gone overboard with that is Jacob Lurie, he has practically single handedly written down a foundational work of higher algebra and algebraic topology, which essentially precisely corresponds to what you are asking. How does mathematics look like if we weren't constrained to 2d paper to write down equations. Of course to actually write down those things you have to somehow project the equations to several essentially 1d sections or invoke very abstract methods to reason about them. Of course algebraic topology already has tools for that.

The early impetus came from Grothendieck I believe, who began to create a theory of "higher groupoids" in several thousand pages, ordinary groupoids being categories in which every morphism is an isomorphism.

Extremely nice, very out of the box.

On TV in the Netherlands we have a program that aims to educate the public (state sponsored tv), every couple of months there is a new episode, they are listed here (but they are in Dutch): http://dewerelddraaitdoor.vara.nl/DWDD-University.2772.0.htm...

So far they were about the big bang, the quantum world, Einstein (All from Robbert Dijkgraaf, now director of the Institute for Advanced Study in Princeton), The cow and silicon valley. They were always set in a large theater (with live public) with a stage full of props (Inflating balloons to represent the inflating universe, buckets of sand to represent the amount of stars in a galaxy, cameras on rails to represent relativity, living cows as well as all their nice parts of meat on display.) These "presentations" are very nice to watch and my wife who is not a scientist always enjoys them as well.

There are also museums like Nemo in Amsterdam (https://www.e-nemo.nl/nl/ontdek/brein/) and Naturalis in Leiden (http://www.naturalis.nl/en/), these places are filled with rooms with stuff to touch and experiments to perform. I'm sure the US is laced with those as well so it is not that ground-breaking of an idea.

A holo-deck would be the killer feature for this way of communicating.

I thought about Robbert Dijkgraaf's TV lectures as well. A must see! I have no idea if there are any english subs though.

On the topic of how representations relate to programming... if you have ever tried to express some tricky math in code you might have had a sense that it would be more clear if it were written out on paper with the proper notation. That's because mathematicians have streamlined their reasoning by deciding one day, well, I use the sum function a lot, let's call it Σ. Or I'd like to be able to recognize integrals from a mile away so let's call it ∫. Not to mention those symbols come along with a spatial organization: the lower limit of integration goes below, the upper limit goes on top.

Programmers can't easily streamline like this. Maybe there are multiple forces locking us in: the keyboard, text editor, and programming language. Sure, maybe the optimal way to write code is with a string of ascii characters arranged left to right top to bottom but I doubt it. So even in the visual-symbolic space there is unexplored potential.

I'm glad someone is out there tackling these assumptions and much more, I should keep an eye on his work.

On an unrelated note, his website is entertaining: http://worrydream.com/

His website is legend among UX designers because it's such a great example of how to do a portfolio creatively.

Really? I thought he was a celebrity among programmers because of his imaginative approach. Surely any UX designer worth their salt would find his home page to be moronically simplistic in the way it tries to shake up graphic design conventions. He is interesting, but not, I think, in a "design" way. Also, "worrydream", "beast of burden"? A bit too neurotic to be considered appropriate by, well, anyone grown up.

A website meant to be usable is a bit different from one meant to be a portfolio. From that perspective, I think his website works well, but it is a bit dated in terms of aesthetics (not as flat as the current fashion goes).

I don't think Bret Victor's work is really about design (most of the UX designers I know have never heard of him), it is about invention and is a bit more on the programmer side. Putting him in the designer bucket is too inaccurate (but buckets are always inaccurate).

I saw this talk in person at SPLASH; the puppy in cage for life really pulls at some heart strings. But it didn't really do it for me like the previous talks did (though he didn't have any prototypes to show off as with the previous talks).

My comment was shorthand for:

I discovered his portfolio a couple years ago through: http://www.quora.com/What-makes-a-great-UX-Design-portfolio

and since then it's come up in conversation with UX designers as a cool portfolio 3 or 4 times. But the points you make are valid.

> such a great example

Are you being serious?

Looks interesting, going to watch rest later. I wonder if the graph of trade with England is really the 'basis' for all scientific graphs of data. Didn't Descartes develop the idea of graphing Y in relation to X?

http://en.wikipedia.org/wiki/Cartesian_coordinate_system

great talk! one touchstone i wish he would have embraced is woodworking. having listened to the talk, i can see how woodworking is experienced through many channels: tactile (hand planes, saws), visual, spatial, symbolic (plans, designs), auditory (sonic feedback tells you a lot of about the wood, and how well a tool is working), environmental (the workshop)... i feel like adopting his channels perspective helps me understand why that activity is so engaging. certainly gives credence to his claims, too.

I think something like ipython can get to where he wants to go. I have already used the RISE profile and MDTraj so I can show people my notebook in slide layout, and pull protein models directly from PDB (Brad talks about the provenance of the data). So I can run code live, manipulate meaningful models live. I think some solid navigation wouldn't be too far off. Now we just need holograms. Cheap blue lasers, people. We need cheap blue lasers.

Still watching the presentation, but virtual reality is the medium that will eventually allow for a lot of the things he's talking about.

Edit: he later dismisses this, but I don't think he's right to do so.

VR still lacks good tactile, kinesthetic representations. Not to say it's not a step in the right direction, but it is arguably taking a deeper step into visual-only representation.

(edit: relevant link, Victor's rant on hands in interaction design. http://worrydream.com/ABriefRantOnTheFutureOfInteractionDesi...)

While it's neither perfect nor complete, I can say from multiple hands-on experiences that the Tactical Haptics grip works much better than you might expect.

https://www.youtube.com/watch?v=XBB_OFMJ-Go

I've never used them personally but I'm sure they do improve the experience. I suppose there's a point at which that's still tricking the human nervous system in ways that a physical model doesn't have to, but haptic VR is also more realistic with today's technology than dynamically self-assembling nanobots.

He dismisses it as an end, but VR is the beginning...

exciting to be young and engaged in technology at a time like this

Reply on news.ycombinator.com