Turning a MacBook into a Touchscreen with $1 of Hardware (2018)
anishathalye.com
anishathalye.com
I really wouldn't have expected this to work but once you read about it.... of course it would.
This is one of those things that while there maybe wasn't a 'problem' shows how knowing a few things critical, you can really solve problems / come up with solutions that nobody would have ever thought of (well I wouldn't have...).
It's a fun DIY project and you can get it almost working. I suspect edge cases (stemming from different lighting conditions and imperfect finger detection) would render this method unworkable in general.
[1] https://www.macobserver.com/imgs/tmo_articles/20100129ipadbo...
The bonding process adds a bit more cost.
It's all been done before.
Again, fun DIY project, limited commercial value.
Gestural stuff is nice when it's transparent and guess-able. Sadly not often the case, but when it is it's pretty magical.
I tried doing this with the internal webcam, using a clip-on fisheye lens, and mirrors. And eventually punted, using the bare webcam only for head tracking, and adding usb cameras on sticks perched on the laptop screen. With more sticks to protect the usb sockets from the cables. And lots of gaff tape.
> leapmotion
Leap Motion has finally been acquired, so the future of the product is unclear. And it's Windows-only (the older and even cruftier version supporting linux, doesn't do background rejection, and so can't be used pointing down at a keyboard). But it has apis, so you can export the data. My fuzzy impression is it's not quite good enough for surface touch events, but it's ok-ish for pose and gestures. When the poses don't have a lot of occlusion. And the device is perched on a stick in front of your face.
> Gestural stuff is nice when it's transparent and guess-able. Sadly not often the case
I fuzzily recall some old system (Lisp Machine?) as having a status bar with a little picture of a mouse, and telling what its buttons would do if pressed. And a key impact of VR/AR is having more and cheaper UI real estate to work with. So always showing what gestures are currently available, and what they do, should become feasible.
Even on a generic laptop screen, DIYed for 3D, it seems you might put such secondary information semitransparently above the screen plane. And sort of focus through it to work. Making the overlay merely annoying, rather than intolerable.
But when it all works, yeah, magical. Briefly magical. The future is already here... it just has a painfully low duty cycle. And ghastly overhead.
But once it it got warm and humid it was unusable. I had to turn off the touchscreen and then I could at-less use the mouse.
This hack would work even when the air is humid.
The only thing I don't get is why Apple didn't create MBP touchpads compatible with their digitizer. When I saw the announcement of the first Macbook with this new huge touchpad that seemed so obvious to me, and I was baffled they only went for that strange touchbar.
I use a mouse most of the time but when I have to press buttons on dialog boxes it's easier to raise my hand from the keyboard and push the button. Same for touching a window a raise or above the others.
The size of the button or the exposed size of the window make the difference. There must be plenty of space or mistouches kill the experience. Resizing windows is something I do only with the mouse.
A hybrid approach, mouse/touchpad and touchscreen is the best IMHO. If my next laptop has a touchscreen option I'll buy it.
* Not counting the "Metro" monstrosity that briefly reared its head during Windows 8; I'm not sure if it's still around.
Of course even 4 years after the Windows 10 launch you still have the new UWP settings app and the old Control Panel. It's a bit of a mess.
I was delighted to see this is so much better. Figuring out touch events from the reflection is really cool. Well done.
You can make junkyard bound stuff do amazing things when you program them right, but you pay for that programming or time spent.
Or to put it another way you wouldnt invent a new way to desalinate seawater and then say the cost is $10/litre instead of 2 cent because of r&d on the device you made..
Depends on the marketing strategy, most likely. But I like GP's point about undervaluing programmer time. It is non-free and everyone down to the programmers themselves tend to grossly underestimate its cost.
Humans are kind of expensive to keep alive and happy enough to program willingly
/That last statement had more caveats than I wanted...
If I paint my wall, I would generally say it costs the price of the paint, not whatever I'd have to pay to have someone do it for me.
Or what about the people who tinker with their own smart home solutions with arduinos and raspberries. If you factor in the price of work/research/coding, it would be very hard to justify not buying an off-the-shelf solution instead.
That's the mentality I'm challenging here. Mentally you don't account for the cost of your time, but from an outside perspective, you're spending it all the same.
That you have to factor in the price of labor with DIY electronics is the main reason you don't see more of those easy electronics projects available as some off-the-shelf contained product (where's my IoT API-accessible thermometer for $10!!!???).
And then it gets turned around and copyrighted/patented anyways.
So yeah it should cost $.02/gal . It's only cause of profittering goons at every level do essential needs for humans get privatized and 'owned'.
Maybe some time-rich person will fork it and do so.
What makes this one different in this regard?
(I'm not trying to say it's a meaningless project.)
My friends and I built a time machine radio (i.e. you tune it to a decade, with a knob and everything, and radio news from the decade was included if we could find it) and won best hardware hack, so it wasn't a total loss. Had a blast either way.
(Not a comment on you, but you poked the button)
While my hackathon experience is limited, I think any attempt to crown a "winner" reduces the experience. Of course, I am too sensitive because everyone can just enjoy it...but as soon as there's a winner there starts to be rules that people try to game, the are added restrictions and limits that reduce what is being created, etc. And creating is the point.
The problem I think is that judges themselves are incompetent/careless — when you promote a “bad” winner, you just discourage everyone from trying, because they realize there is no value to this kind of promotion; its bullshit.
I had a similar experience in my university — some kind of business project idea competition; we submitted my lab’s current project, and there were other interesting projects involved who I would have been happy to lose to.
Instead I lost to the girl with the contentless idea for edible spoons for developing countries... an idea I’d seen repeatedly in recent news cycles because some NGO in India was doing it for the last three years.
Five minutes of googling would have discovered this, but instead we simply ignored the competition since.
Attending itself was valuable, along with the half-attempt at winning — we clarified the project and ideas, and had an excuse to clean things up — so it would have been a good exercise to do yearly (and probably for the other projects involved), and the competition was healthy.
But it just takes an incompetent judge to spoil the whole thing.
There’s nothing wrong with people trying to game the competition, its a natural function of competing. But the judges are meant to operate as experts, and an expert should have (some) ability to discern bullshit from something real.
I've become extremely skeptical of the hackathon model in general. It doesn't really produce good projects or products, it encourages an unsustainable style of working and it's a terrible place to recruit good developers. It can encourage people to build stuff, but only monkey patched, bursting with buzzwords, overly flashy projects that are thrown away at a moment's notice.
I actually did something similarly stupid in college; for an embedded development class, my team needed a final project. I proposed a hat for the blind, with distance sensors and buzzers on the inside to inform distance from the walls and such. We also used black conductive thread on a black hat, so it was impossible to work with except for the girl who threaded it in the first place.
Easily the professor's favorite project, and I guess he still uses it as an example for future projects (and at some point he had some team extending it into some kind backpack?)
Later a friend working at a retail store selling products for the disabled contacted me about the project; apparently his boss was interested.
What no one acknowledged is that blind people are not walking into walls... the cane is far more practical, efficient and effective tool than this thing could ever be.
I had meant it to be a joke.
The best thing about it is, the people who need and want the device aren't the ones walking around reading their phone, it's the ones who can't help but mind someone else's business - and this merely feeds their need rather than satisfies it. So there will be an never-ending market of unsatisfied busybodies.
Rewriting it in C/C++/Rust instead of Python would be one... Any ideas?
It just takes some work.
Source: have used used webcam cube cams in a robotics application where latency needed to be accounted for.
But all kinds of post-processing can be (and is) done on readout (pixel by pixel) without needing to keep the whole frames and go through them. You just keep some calculated parameters from previous frames (and not the whole frames).
You can do a lot of processing this way, incl. scaling, white balance, color correction, effect filters, etc.
Web cams add usb interface and an additional controller chip, so maybe there's some added latency there. But if you use the camera sensor directly over CSI, you can get pretty low latency.
Also depending on the bus being used cameras often compress the image using JPEG to reduce the bandwith (very common with USB cameras, I doubt that the Mac camera does that).
Then on a computer you have the latency of whatever pipeline is being used and how much buffering is involved. For instance since typically the display doesn't run exactly synchronized with the camera you'll have some frames that'll end up "stuck" waiting for the monitor's vsync. If the app then does some internal buffering on top of that you can very easily reach a noticeable amount of latency.
In general you can consider than anything above 100ms is easily noticeable and that's only 3 frames at 30fps and 6 at 60, it's really not that much when you consider the complexity of modern video capture pipelines.
A pipeline might look something like demosaicing -> denoising -> light/exposure adjustment -> tone mapping -> encoding
Some of these steps will likely have hardware support (e.g. demosaicing, encoding) some won't. You may carry a few frames around for denoising, e.g. You have to sync up with the display at some point. Some of the steps might change order too, depending on what you are trying to do and what hardware support you have.
I haven't looked at available hardware specs recently. You used to be able to get very basic capture cameras that offloaded nearly everything to the computer - so that would give you minimal latency on the capture side (limited by your communication channel obviously); However keeping a stable image as resolution and frame rate grow is really hard that way, eventually impossible without a realtime system. More modern cameras are going to do much of that on board.
The alternative that usually gets used in productions with a moderate budget is a hardware camera with HDMI or SDI output into a Blackmagic capture device. There's still some latency, but it's better than a webcam. It's pretty much down to the minimum of what you can do with a computer -- the capture device still has to buffer a frame, transmit it to the computer, which copies it into RAM, then blits it to video RAM, where it waits for the next vertical refresh.
All that said, the human finger doesn't travel all that fast, so for a HID application like this, you could probably get the latency way down by extrapolating the near-future position of the finger, much like iPadOS does with Pencil.
You don't need memory to have latency. Speed of light / electrons is enough...
That said, webcams absolutely have extra processing (and thus latency).
Would it? It's using libraries for the heavy lifting already.
We could have a better/safer C but alas we don't. It's either C or all the way to Zig/Rust/etc territory (or using C++ like C).
The laptop hardware is different of course, but it would be an interesting performance comparison.
Here's an interesting overview of using Core frameworks with Python: https://developer.apple.com/videos/play/wwdc2018/719/
Though that does not take account of the latency of the webcam itself.
And additionally you’ve made your MacBook a lot less portable.
[1]: https://www.sonnettech.com/product/egfx-breakaway-box.html
They are not asking what kind of problems having a touch screen on a laptop would solve.
> With some simple modifications such as a higher resolution webcam (ours was 480p) and a curved mirror that allows the webcam to capture the entire screen, Sistine could become a practical low-cost touchscreen system.
Found this Kickstarter project that’s now a company. http://serafim-tech.com/products/serafim-keybo-world-s-most-...
I'm not at all interested in this as a keyboard per se, but the fact that it functions as a piano keyboard as well gives me ideas. I wonder if the latency and resolution would allow for things like automatic transcription of written documents as you write, a mouse replacement, or other cool things that I haven't thought of yet :)
They claim they have an SDK. I'll have to explore this some more when I have time later this evening.
1. Point a camera at a surface (be it a piece of paper or whatever).
2. Write/print a word on that paper and use CV/OCR to parse it.
3. Detect when a person has pressed on that word and exec the command specified by the text.
Instant macro pad?
Trying to avoid the wacom/tablet path to keep it lightweight :)
[1] - http://www.proelite.co.in/Amazon-Kindle/All-Amazon-Kindle-10...
Als interested to see how it works with different lighting and how dirty the screen can get before recognition fails because the reflection is too „muddled“ ;)
Either you have a mirror that converts your webcam into ultra wide angle so you cover the top edge of the screen (which you pay for by poor resolution on the bottom) or you decide not to cover the whole top area (which is where most window control elements are).
The video action all happens on the lower half of the screen for a reason.
A solution would be some sort of orthographic lense, that basically splits the sensor into multiple distributed lenses to avoid the FOV problems. This however would be not trivial to get right without serious tools.
On the topic of touchscreen Macbooks, I've been using my Mac a lot less since I've spent a lot of time on a touchscreen Windows laptop. It is super handy! A lot of times when using the Mac I'll absentmindedly drag something on the screen only to be disappointed. Apple's resistance to the idea reminds me of their stubbornness on the single button mouse. Admitting you're wrong is not a weakness.
They sell iPads that are touch-enabled, that work well with touch.
However I do strongly feel that touchscreen is the opposite to productivity, maybe usability to the end users, but definitely not what developers need/want.
I agree it's less productive on its own, but it works well when used as a hybrid where you have two types of pointers on one screen.
I may not prefer touch screens but they can be a complete game changer for some users.
Vertical touch screens trade the more traditional wrist/elbow strain/RSIs for other arm/elbow/shoulder strain/RSIs (including things like "gorilla arm").
Given the nature of repetitive stress, the growing consensus seems to be not that one orientation is better than the other, but that a range of orientations, and a mixture of movement types / input methods is better. For instance, move your arm some to touch a vertical screen, to mix things up from mouse movements and keyboard movements.
Consider if the authors of the article had used (for example) Ruby - you probably wouldn't ask why they didn't do it in Python3, so why do you do it if they write it in Python2?
Python 2 will be EOLed soon.
4 months 25 days 5 hours left until Python 2 is retired. After that point it will no longer be maintained by the Python project.
Of course Linux distros and companies will keep Python 2 running for years to come because of all of the tools that are Python 2 only.
New projects ought to be written in Python 3. However, in this case my guess would be that OP probably used Python 2.7 because macOS ships with Python 2.7 preinstalled.
So if we could, Apple are the ones that we ought to speak to. Convince Apple to ship macOS with Python 3 pre-installed.
I'd have hard time finding support for some Ruby 1.8 upstream dependencies for recent projects, so I don't agree with you.
This is especially true of MacBook screens!
Anything that requires tons of visual calibration is serious work. Both for the programmer and the computer.
It worked for a planned demonstration, but I imagine this would be impossible for Mass production. And I wonder if the users found it useful due to latency and accuracy problems.
They hacked something cool together, I suspect that is the point, and it is awesome.
This isn't a product.
>If a story has had significant attention in the last year or so, we kill reposts as duplicates. If not, a small number of reposts is ok.
That post is from over a year ago, and didn't get much traction.
tbh Reddit's repost detector causes more problems than it solves (it still triggers if the original post was removed).