HNHacker News
TopNewBestAskShowJobs

smacke

466 karma · joined February 9, 2019

https://github.com/smacke
submissionscomments
smacke··on IPyflow: Reactive Python Notebooks in Jupyter(Lab)
For this particular project, Python is a requirement.

For the general approach, the answer is more complicated. It depends on what hooks the language implementation exposes -- and even if it exposes enough to make this work in theory, tracking dataflow at the same level of accuracy and granularity as IPyflow does may not be possible without taking an unacceptable performance hit, or without sacrificing portability across language versions.

My hope is that the approach can scale to languages like Julia or R, but I'm not as familiar with those languages as I am with Python, and I kind of suspect each language may require its own bespoke tricks.

Regardless, for Python it was a journey roughly 3 years in the making (and still ongoing) -- other languages would be easier now that I've learned a fair amount, but the work to add this kind of support is by far the most complex I've ever done.

smacke··on IPyflow: Reactive Python Notebooks in Jupyter(Lab)
Thank you for all the feedback, positive and constructive! Yep docs definitely need a lot of polish :)

3. I actually started from scratch -- ipyflow's reactivity model is a bit different from these, since for Python, my experience is that static dependency inference is too unreliable to be useful. (Though after talking with the Pluto maintainer earlier today it sounds like Pluto may be reaching some of the same conclusions and also be moving toward a dynamic dependency inference strategy)

In the future, my hope is that as a community we will develop a live-coding analogue to lsps which one might call a "language kernel protocol" so that we can standardize some of these features across different languages / editors

4. For top-level / module-level statements, yes there is lots of overhead (> 100x), but it's largely limited to those statements (i.e. external library calls, recursive function calls, etc have close to 0 overhead thanks to intelligent instrumentation disabling for these) and turns out to be OK in practice (more details in nbslicer paper https://smacke.net/papers/nbslicer.pdf). At some point I'll run it through a profiler and try to grab the low-hanging optimizations but it hasn't been noticeable so far.

Surfacing the DAG is definitely something I want to do at some point; we have all the information in the backend so we should try to surface it in the frontend.

smacke··on IPyflow: Reactive Python Notebooks in Jupyter(Lab)
It's running on top of vanilla CPython, but with heavy instrumentation via sys.settrace as well as ast transformations. x and y are just normal Python ints. The downside is the overhead, but it's paid mainly for top-level / module-level statements, so I've found it to be acceptable in practice. The benefits are on portability -- as it scales for all the major Python versions supported today (3.6 to 3.11), and even for some different Python runtimes such as Cinder.

More details in this paper: https://smacke.net/papers/nbslicer.pdf

But overall I agree I need to get on top of the docs and talk in more depth about the implementation there.

smacke··on IPyflow: Reactive Python Notebooks in Jupyter(Lab)
Yep. I think there is also a way to enable it by default in your ipython profile which I'll document at some point, so that you don't have to run `%flow mode reactive`. I'm curious though -- personally I much prefer to use the opt-in reactive execution mode with ctrl/cmd+shift+enter; curious to understand your preferences better :)
smacke··on IPyflow: Reactive Python Notebooks in Jupyter(Lab)
Hi -- author here. I just presented this work at JupyterCon 2023 so I figured it was time to advertise more broadly. There are a few rough edges but my hope is that, by making all the reactive behavior opt-in and only enabled for in-order execution (i.e., cells above the one I execute will never reactively execute by default), it can be predictable enough to be useful in practice.

There's still a long way to go to get e.g. full dataflow understanding of all the common libraries, understanding file paths, autoreload integration, etc., but after nearly 3 years of on-and-off development I think it's finally useable-ish.

smacke··on Ask HN: What have you created that deserves a second chance on HN?
https://github.com/ipyflow/ipyflow

IPyflow is a new Python kernel for JupyterLab that understands how variables and cells depend on each other, making it easier to reason about notebook state. It adds opt-in reactivity, so that pressing ctrl+shift+enter triggers execution of all cells that depend (recursively) on the current cell. Furthermore, with its `code` function, you can see exactly what code is needed to reproduce a given variable. I started working on it after watching the famous talk "I Don't Like Notebooks" by Joel Grus, and, anecdotally, I like notebooks just a little more when I use this kernel :)

smacke··on Monumental (if correct) advance in number theory posted to ArXiv by Yitang Zhang
Probably a reference to Yongfei Ci actually (https://dailyillini.com/news-stories/2014/06/20/yongfei-ci-s...)

UIUC is not having a great track record wrt grad student murderers o_O

smacke··on The Jupyter+Git problem is now solved
> It would be nice if there were something that would make out-of-order problems light up, the way that code editors can highlight errors while you're editing.

Check out https://github.com/nbsafety-project/nbsafety

smacke··on My lab went from 4000 kg to 130 kg of waste a year
I think this depends. Universities seem to have the power to generate "tuition waivers" that accompany TA positions and a few external fellowships such as NSF GRFP, but if you're funded solely from one or more of your advisor's grants, I believe that, at most places, the grants are actually used to fund the full cost of tuition. And as we all know, tuition is kind of insane at most places due to bloated admin.

Source: what I remember my advisor telling me when I was a grad student

smacke··on What if Planet 9 is a Primordial Black Hole? (2019)
If that thing were to appear on Earth, wonder how long before it becomes 6M⊕.
smacke··on Launch HN: Deepnote (YC S19) – A better data science notebook
For A), I'll just leave this here :)

https://nbsafety.org

smacke··on The NBA bans customers from putting ‘freehongkong’ on customized league jerseys
I probably should have phrased it differently. I think the attitude in China is that, if somehow forced to choose between individual liberties and economic prosperity, many will choose the latter -- because they see it as enabling the former more capably than any sort of constitutional rights.

Btw I'm intentionally trying to stay take a neutral stance on this topic when discussing my understanding of attitudes in China. In fact my personal viewpoint is that economic prosperity without a strong and healthy respect for human rights will have bounded lifetime, but that's a different discussion. Regardless I think it's important to understand all the viewpoints.

smacke··on The NBA bans customers from putting ‘freehongkong’ on customized league jerseys
They are a bit unregulated, and people warned me not to eat there too often. For example, I was told that people have died of rat poison after eating rat meat (for rats apparently captured from the street). Another issue is that the vendors would sometimes reuse their cooking oil (so-called "gutter oil"). That said, I got the impression that it was OK to indulge on the occasion, and it is true that the atmosphere is really great.

Ironically, these night markets are disappearing a bit as prosperity goes up. For example, in Shenzhen, a lot more roads were built to accommodate the huge increase in the number of automobiles, and it seems that many of these markets have succumbed to the concrete jungle.

smacke··on The NBA bans customers from putting ‘freehongkong’ on customized league jerseys
American here. I spent a couple of months in China and have Chinese friends, so here are a few things off the top of my head (take with a grain of salt and consider talking with Chinese people for a more authentic perspective):

- Reliable high-speed railway.

- Night markets -- lots of people socializing in the street into the early hours of the morning, enjoying street food, beer, etc.

- A sense of shared experience that's difficult to find in hyper-individualistic societies.

The Chinese perspective, AFAICT, is that individual liberties are worth trading for prosperity and so-called "economic" liberties. I'm told that this is something that's difficult to understand unless you've experienced the sort of progress that China has observed firsthand.

smacke··on Ask HN: Side projects that are making money, but you'd not talk about them?
I built a small library for synchronizing subtitles with video (https://github.com/smacke/ffsubsync) at a hackathon, and a couple of users appreciate it enough to sponsor me for $11 / month, which comes to $22 / month after Github matches contributions :)
smacke··on China’s 'Thought Transformation' Camps [video]
I don't think anyone is forcing Kantian ethics on the CCP. If they are, it's not working very well. :)

But I'm glad we can trace some disagreements regarding whether specific actions are ethical back to differing ideologies. If we can abstract some of these ideas it should be easier to have conversations that are a bit removed from emotional factors.

smacke··on China’s 'Thought Transformation' Camps [video]
Based on the footage, and even in spite of the agenda made clear from the interviewer's line of questions, it seems difficult to deny many aspects of the BBC reporting. These institutions do exist, and are targeted at the Uyghur minority. My current internal bias suggests that enrollees at these institutions would be punished for expressing anything but positivity regarding their situation. When one group of humans has power over another, history and even academic exercises [1] suggest that things get ugly.

Much of the western world is influenced by Kantian ethics [2], of which a central tenant is that the end (potentially greater societal stability, in this case) is not justified by the means. This system of thought does not seem to hold much sway by the CCP, and I believe many disagreements can be traced back to this point.

[1] Google "Stanford Prison Experiment" [2] https://en.wikipedia.org/wiki/Kantian_ethics

smacke··on China’s 'Thought Transformation' Camps [video]
All those things you mentioned are a) bad and b) off topic. I suggest you create a new post to discuss those.

The right vs wrong question seems straightforward. If the people in these concentration camps are suffering, then it is wrong. If they are unable to speak about whether they are suffering, then that is also wrong.

It's only a "story planted in [your] mind" until it happens to you.

smacke··on Tiananmen Square: China minister defends 1989 crackdown
As of this writing, this post is 2 hours old, ranked #96 (4th page of HN), and has 117 points. I believe it was on the front page 30 minutes ago. This is a steep drop in ranking -- what are the factors in the HN ranking algorithm that could contribute to this?
smacke··on Tiananmen Square: China minister defends 1989 crackdown
Hi, you are probably being down voted because your post is being interpreted as "whataboutism", although perhaps that was not your intention.

https://en.m.wikipedia.org/wiki/Whataboutism

smacke··on Better Bash History (2012)
You might enjoy checking out 'fasd', which, among other things, creates an alias ('zz') with autocomplete functionality for changing to frequently-used directories.

https://github.com/clvv/fasd

smacke··on Gene Therapy Was Hailed as a Revolution – Then Came the Bill
Playing devil's advocate: there may be an argument that such care would not have been developed without the free-market monetary incentive. (Maybe not; maybe it would have been developed thanks to some government-funded grant. The point is: it's worth understanding what's possible and impossible in our current system.)
smacke··on Can a Person Learn While Sleeping?
This is the first time I had heard of this rigid lattice problem and I thought it sounded very interesting; if anyone else is interested, here's a link to some details: http://datagenetics.com/blog/november12014/index.html
smacke··on Facebook Still Tracks People on Yelp, Duolingo, Indeed
> So if you can't have perfect privacy, you'll have nothing at all? That's quite an all-or-nothing way of looking at this issue.

I don't think that's the point; I think the OP is saying that in OP's particular case, the marginal benefit of Facebook privacy is much smaller than the marginal benefit of privacy in other aspects of life, and the lack of privacy from these various other aspects seem to be here to stay, therefore why worry about something relatively trivial like Facebook?

There may be other fallacies in OP's assumptions of course, but those notwithstanding I agree with the underlying logic. E.g. if I deem riding motorcycles an acceptable risk, then arguably I shouldn't spend energy worrying about whether my airplane will crash.

smacke··on Show HN: Automatically synchronize subtitles with video
Agreed -- definitely worth thoroughly investigating this space to see whether there are any ideas that can be borrowed.
smacke··on Show HN: Automatically synchronize subtitles with video
You are most welcome! This is exactly the use case I was initially targeting; I got super lucky that the VAD-based synchronization happened to be low-hanging fruit that has a higher "Wow" factor.
smacke··on Show HN: Automatically synchronize subtitles with video
Sorry, I forgot to answer your last question. It turns out that, because of how the algorithm works, the number of blocks shouldn't matter. Since it is discretizing time windows in 10ms increments, the granularity of the "effective blocks" is small enough that putting two separate large blocks on the screen, each for a shorter period of time, is roughly equivalent to putting a single large block on the screen for twice as long (for synchronization purposes, that is).
smacke··on Show HN: Automatically synchronize subtitles with video
This project came up when I was researching this idea. It looks especially useful because the alignment algorithm is less restrictive (can introduce splits/breaks), so I plan on taking a closer look for sure. Given that it does not do any speech detection and requires a reference subtitles file, I'm not sure how well it would work for aligning the encoding for the speech-induced part, which will have a lot more noise. Particularly, the linked project works by aligning time intervals [a,b], while subsync is "dumber" in that it discretizes things into binary strings, and an interval constructed by merging consecutive 1's and 0's in speech detection-induced binary strings will probably be a lot harder to match to the subtitle intervals.

Either way, I also thought this was a cool project and will be thinking more about whether any ideas can be borrowed.

smacke··on Show HN: Automatically synchronize subtitles with video
You are most welcome, although unfortunately it looks like this project doesn't support your use case yet. Right now it can only correct "constant drift", although I'll definitely be thinking more about other scenarios based on the reception.

That said, it sounds like for your particular case, if you use VLC, you might be able to adjust the subtitle FPS so that you don't have to continuously manually resync. (I haven't run into this so just going off what others have mentioned.)

smacke··on Show HN: Automatically synchronize subtitles with video
My guess is that your subtitles don't have a "fixed drift". Unfortunately, this project only works for constant drift at the moment -- dynamic subtitle drift is a harder problem but absolutely worth thinking more about for future work.
← PreviousPage 2 of 3Next →