HNHacker News
TopNewBestAskShowJobs

hundredwatt

681 karma · joined January 29, 2009

email: <username> at gmail Twitter: @jasonnochlin

[ my public key: https://keybase.io/hundredwatt; my proof: https://keybase.io/hundredwatt/sigs/sAhvC-zN-G30cPShxbXA7UC-0MmpSWpit0N0CNOtDME ]

submissionscomments
hundredwatt··on Extending That XOR Trick to Billions of Rows
> To be pedantic, not guaranteed. The xor of multiple elements may erroneously have a passing checksum, resulting in an undetected false decode

The false decodes can be detected. During peeling, deleting a false decode inserts a new element with the opposite sign of count. Later, you decode this second false element and end up with the same element in both the A / B and B / A result sets (as long as decode completes without encountering a cycle).

So, after decode, check for any elements present in both A / B and B / A result sets and remove them.

--

Beyond that, you can also use the cell position for additional checksum bits in the decode process without increasing the data structure's bit size. i.e., if we attempt to decode the element X from a cell at position m, then one of the h_i(x) hash functions for computing indices should return m.

There's even a paper about a variant of IBFs that has no checksum field at all: https://arxiv.org/abs/2211.03683. It uses the cell position among other techniques.

hundredwatt··on Extending That XOR Trick to Billions of Rows
The graph constructed by using bloom filter-style hash functions supports a decoding process called "peeling" where you:

1. Find a batch with 1 missing element 2. Delete that element from its other assigned partitions 3. Repeat, as the modified batches may now be recoverable

This iterative process (surprisingly!) succeeds with very high probability as long as the number of partitions is 1.22x larger than the number of missing elements with k=3 hash functions.

hundredwatt··on Extending That XOR Trick to Billions of Rows
You don't lose absolute guarantees, but the probabilistic nature means the process may fail (in a guaranteed detectable way) in which case you can try again with a larger parameter.

The "bloom filter" name is misleading in regard to this.

hundredwatt··on That XOR Trick (2020)
A neat trick to make the accumulator both collision-resistant and self-diagnosing.

  For every normalized link id x:
      y = (x << k) | h(x)   # append a k-bit hash to the id
      acc ^= y
If acc is zero, all links are reciprocal (same guarantee as before).

If acc is non-zero, split it back into (x', h'):

* Re-compute h(x').

* If it equals h', exactly one link is unpaired and x' tells you which one (or an astronomically unlikely collision). Otherwise there are >= 2 problems.

This has collision-resistance like the parent comment and adds the ability to pinpoint a single offending link without a second pass or a hash table.

hundredwatt··on A faster way to copy SQLite databases between computers
The recently released sqlite_rsync utility uses a version of the rsync algorithm optimized to work on the internal structure of a SQLite database. It compares the internal data pages efficiently, then only syncs changed or missing pages.

Nice tricks in the article, but you can more easily use the builtin utility now :)

I blogged about how it works in detail here: https://nochlin.com/blog/how-the-new-sqlite3_rsync-utility-w...

hundredwatt··on Ask HN: What are you working on? (February 2025)
It’s meant to run in or alongside the pipeline continuously.

Not planning to open source, working on a commercial offering but haven’t launched anything publicly yet.

Would love to hear any more thoughts on the concepts here or my email is in bio

hundredwatt··on Ask HN: What are you working on? (February 2025)
I'm building a new tool for end-to-end data validation and reconciliation in ELT pipelines, especially for teams replicating data from relational databases (Postgres, MySQL, SQL Server, Oracle) to data warehouses or data lakes.

Most existing solutions only validate at the destination (dbt tests, Great Expectations), rely on aggregate comparisons (row counts, checksums), or generate too much noise (alert fatigue from observability tools). My tool:

* Validates every row and column directly between source and destination * Handles live source changes without false positives * Eliminates noise by distinguishing in-flight changes from real discrepancies * Detects even the smallest data mismatches without relying on thresholds * Performs efficiently with an IO-bound, bandwidth-efficient algorithm

If you're dealing with data integrity issues in ELT workflows, I'd love to hear about your challenges!

hundredwatt··on Ask HN: Books about people who did hard things
+100000
hundredwatt··on Kuvasz-streamer: open-source CDC for Postgres for low latency replication
That’s one of the cases where query-based CDC may out perform log-based (as long as you don’t care to see every intermediate change that happened to a row between syncs)
hundredwatt··on A High-Level Technical Overview of Homomorphic Encryption
> Much of the FHE industry is focused right now on the subset of applications where legal hurdles make the two available options “slow, but compliant” and “fast, but incurs prison time.”

Anyone have any examples of these applications?

hundredwatt··on Ask HN: What does your BI stack look like?
+1 for Metabase

For our team, using an ELT architecture (as opposed to ETL) [1] for managing our data warehouse has greatly reduced the complexity of our data processes. Instead of creating ETLs for every table we want to load into the data warehouse, we create the minimum necessary setup to copy the table into our data warehouse. Then, we write transforms, which are simply SQL statements, to generate wide-column tables that our non-technical users can use to explore data without worrying about joins or having to learn esoteric naming conventions.

Custom EL Scripts -> Redshift -> Transform Statements -> Redshift -> Metabase supports the data needs of all our departments with no dedicated data team members.

[1] https://www.dataliftoff.com/elt-with-amazon-redshift-an-over...

hundredwatt··on Ask HN: The habit adopted in 2016 that had the greatest impact on your health?
I had the same problem. In addition to regular exercise, try doing these stretches for once or twice a day: https://www.youtube.com/watch?v=FdNS95hpL-o (they are fun too! Perhaps get a group together at your office to stretch together once a day)

These exercises work to move your body back toward perfect posture, undoing the damage caused by sitting, typing, etc.

I stopped using these in November due to travel. My back and shoulder pain returned. It took about 7 days of consistent stretching for the pain to go away again.

hundredwatt··on Hacked: commit to rails master on GitHub
I threw together a quick 'n dirty Rails generator that will generate the code for white/black listing all model attributes with attr_accessible/attr_protected.

Here's the file: https://gist.github.com/1975167, just add to lib/generators in your Rails 3 app, then do rails g mass_assignment_security -h

Hopefully others find this helpful

hundredwatt··on Ask HN: Freelancer? Seeking freelancer? (December 2011)
SEEKING FREELANCER - Remote - GaggleAMP.com

GaggleAMP is hiring part-time software developers and UX designers to help us extend our social amplification platform. On the frontend, we use jQuery and HTML5/CSS3 via HAML templates. Our web application's backend stack is Ruby on Rails 3 with MySQL and Redis.

We'll consider hackers with any experience level, intern and up. If interested, send an email with a brief bio and one or more links to past work to jason AT gaggleamp DOT com.

hundredwatt··on Ask HN: Who is Hiring? (December 2011)
Remote - GaggleAMP.com

GaggleAMP is hiring part-time software developers and UX designers to help us extend our social amplification platform. On the frontend, we use jQuery and HTML5/CSS3 via HAML templates. Our web application's backend stack is Ruby on Rails 3 with MySQL and Redis.

We'll consider hackers with any experience level, intern and up. If interested, send an email with a brief bio and one or more links to past work to jason AT gaggleamp DOT com.

hundredwatt··on Ask HN: Freelancer? Seeking freelancers? (October 2011)
SEEKING FREELANCER web design/UX, remote

GaggleAMP is looking for freelance web designers to work on a per project basis.

Projects will range from add dynamic elements to landing paged TO creating the UI for new application features.

Ability to code HTML/CSS and javascript is a huge plus

Send portfolio to jason at gaggleamp dot com if interested

hundredwatt··on Ask HN: Should I work for a startup who's sole purpose is to be flipped?
If it is a smart team working on an interesting/challenging problem and you think you would enjoy it, then the fact that they want to flip the business should only be a secondary consideration.

Just remember to set your expectations correctly about cash/equity incentives: http://www.bothsidesofthetable.com/2010/09/06/how-to-discuss..., http://www.bothsidesofthetable.com/2009/11/04/is-it-time-for...

hundredwatt··on Ask HN: Is it feasible to use redis as the only datastore?
Why not?

EDIT: If it meets your application's logical and scaling needs, there's no reason you couldn't use it

hundredwatt··on One of Google’s Self-Driving Cars Gets into an Accident
<i>"While Jalopnik believes that the fender-bender is proof that self-driving cars may not be in the best interests of society, we have a different take. There were 10.2 million traffic accidents in 2008, which results in 39,000 deaths. That’s 17.9 people per 100,000 licensed drivers. If Google self-driving cars can beat those statistics, they could actually prevent more accidents than they create. We also waste millions of hours commuting and driving through traffic; imagine if you had that time to be productive instead."</i>

Also, computers can't get drunk...

I'd love to have these brought to the masses. But, no matter what the statistical benefits are (safer and time saving), I can't picture a time in the near future when normal people will be accepting of automated vehicles. Just like this incident almost was, 1 death caused by a self-driving vehicle is going to overshadow 100,000s of deaths by human-driven vehicles. Even once proved feasible, I can't see something like this being adopted in a matter of years or even decades.

Anyone know of any historical examples of a similarly disruptive, but untrusted technology radically changing people's lives? How long did it take for adoption?

hundredwatt··on How our SaaS startup got 1000+ signups in just 7 days, without getting Crunched.
Don't think of the $15 theme as permanently defining the look and feel of your site. It needs to be just good enough when you're starting to look like you have some legitimacy. If you're startup is going to be generating any amount of significant revenue for you, after some amount of time, you'll be able to go back and revisit the design.

If you're a programer and not a designer, just think of your design as another optimization point.

Whenever I worry about the design/branding of my current startup, I visit the historical timeline of amazon's logo as a reminder that we'll be able to make it better eventually, but there's more important things to work on now: http://www.kokogiak.com/gedankengang/2004/07/amazoncom-logo-...

hundredwatt··on Ask HN: What are some great examples of a Minimum Viable Product?
I said the same thing when I first had the link forwarded to me, but watch the video. The U2 spy plane serves a very important role for the airforce, but can't land or take off on its own.

This is a great example of designing something with an extreme focus on achieving its purpose while ensuring any necessary, but non-core functions are only minimally included.

hundredwatt··on Ask HN: What are some great examples of a Minimum Viable Product?
A friend of mine sent this to me a few years ago:

"""My favorite video on bootstrapping/minimum viable product:

http://www.youtube.com/watch?v=Zv17lF60OHY

It's well worth watching the whole thing."""

hundredwatt··on Groupon files for IPO
I assume there revenue numbers refer to the amount of money collected BEFORE paying merchants. Is that correct?
hundredwatt··on Ask HN: Who is Hiring? (May 2011)
REMOTE/FREELANCER - Boston, MA or remote

GaggleAMP is an exciting company that is committed to bringing a new paradigm to how companies and organizations will use social media.

We are looking for remote or local UX designers and Rails programmers to work on a freelance basis as we continue to build out our existing product and add new features.

Please send resume, portfolio, or GitHub link and a short introduction of yourself to jobs@gaggleamp.com

hundredwatt··on Ask HN: Freelancer? Seeking freelancers? (May 2011)
SEEKING FREELANCER

Boston, MA, but remote work possible

GaggleAMP is an exciting company that is committed to bringing a new paradigm to how companies and organizations will use social media.

We are looking for remote or local UX designers and Rails programmers to work on freelance basis as we continue to build out our existing product and add new features.

Please send resume, portfolio, or GitHub link and a short introduction of yourself to jobs@gaggleamp.com

hundredwatt··on Ask HN: Who Is Hiring Remote Workers? (February 2011 Edition)
I am looking for some PHP developers to work on an app in CodeIgniter and a couple e-commerce stores.

Also, I am looking for a part-time SEO consultant to run a lead generation website and advise a few other projects. Ongoing working, but probably only 5-10 hours per week.

Contact me via email (<hn username> at gmail) if interested. For PHP developers, please send along some example code.

hundredwatt··on Ask HN: Who Is Hiring Remote Workers? (January 2011 Edition)
I would be interested in taking on a remote intern or two over the next few months. I am an independent software developer who works on several client and personal web applications, mostly using Ruby on Rails (jQuery on the frontend). If anyone is interested in internship where they could work with RoR, HTML/CSS/JS, or learn about how a freelance business works shoot me an email (address in profile).
hundredwatt··on Ask HN: Who Is Hiring Remote Workers? (December 2010 Edition)
I am starting a consulting business and am looking to hire subcontractors to work initially on a per-project basis (projects can be as small as a few hours) and potentially on a part-time ongoing basis. Focus is on developing Ruby on Rails applications.

The business is based in Boston, though you can be from anywhere in the world.

I currently need:

- HTML/CSS Designers - Ruby/Rails Developers - QA/Testing

Email me if you're interested (address in profile)

hundredwatt··on Review My App: A Sprite Generator
After seeing DHH's keynote at RailsConf and http://37signals.com/svn/posts/2336-using-css-sprites-with-r..., I decided to try and build a better Sprite Generator. I don't want to bias any potential reviewers, so that's all I'll say for now. If this gets a good response, I'll post some further details in a few hours.
hundredwatt··on Ask HN: My project IsItRails.com — what should I do?
Roland,

This may be a completely useless suggestions as I'm not sure if it's even possible, but perhaps you could port IsItRails to other open-source projects (e.g., IsItDjango). The site could then be a useful way for recognizing open-source projects/devs.

As the database grows, I may start using it show non-technical people that these projects are not just fringe, non-scalable technologies.

Great work!

One last note, as far as fundraising, what's the worst thing that could happen if you sent an email to all the sponsors of RailsCasts asking if they'd be interested in promoting on your site?

Page 1 of 2Next →