HNHacker News
TopNewBestAskShowJobs

NovaX

288 karma · joined May 4, 2011

https://github.com/ben-manes
submissionscomments
NovaX··on Engineers Shouldn’t Write ETL
Is it log-based cdc? The documentation says only 2 dbs are supported that way. We want to replicate customer databases which have schemas outside of our control, with the goal of migrating them to our products. The other vendor schemas might not have your required columns. There are also many flavors of DB2. So very interested but the details are sparse at best.
NovaX··on Engineers Shouldn’t Write ETL
Do you know when DB/2 support will be added (Windows, AS/400)? I've been wanting to switch over from AWS DMS and our in-house support.
NovaX··on IBM acquires Red Hat
I've seen RedHat (via JBoss) knowingly patent work they read in a research paper without referencing or funding them. I was positive it was for a reward, as there was no additionally novel ideas in their descriptions and similar work was already open sourced. I knew it wouldn't hold up so it didn't bother me as impacting my projects, but I was disappointed over the ethics of them doing that. I do wonder how many patent rewards are given for someone else's work by farming from the research community.
NovaX··on Cache Modeling and Optimization Using Miniature Simulations [pdf]
Along your line of thinking, we have a paper scheduled for Middleware (Dec. 2018) and it explores how to make W-TinyLFU adaptive. I can share this paper privately until the conference (ben.manes @ gmail). It shows how to use simple, low-overhead schemes to improve the policy across workloads that exhibit various recency/frequency biases.
NovaX··on Harvard’s Admissions Process, Once Secret, Is Unveiled in Federal Court
The root of this is the Harvard Jew Crisis of the early 1900s, which redesigned the admissions process away from merit. This is the basis of the modern practice at the Ivy Leagues. It was not done secretly, publicized by the university's President in speeches, and had little backlash. It is in fact a result of Asian Americans having many similarities to Jews that they are being discriminated against, but they were not the original target. It will be interesting to see if they were ever explicitly targeted, but so far it appears to be collateral damage as times changed but the process's foundation has not.
NovaX··on Let’s bring back the Sabbath as an act against ‘total work’
Messianic Jewish are generally not considered Jewish by the community, as their beliefs violate basic tenets of the religion. There is a lot of evidence towards these sects being used to manipulate non-religious Jews into conversion to Christianity. They are still welcome, but considered Christians.
NovaX··on Caching Beyond RAM: The Case for NVMe
ARC often wins against LRU, but there is a lot of left on the table compared to other policies. That's because they do capture some frequency, but not very well imho.

You can mitigate the exclusive lock using a write-ahead log approach [1] [2]. Then you record events into ring buffers, replay in batches, and have an exclusive tryLock. This works really well in practice and lets you do much more complex policy work without much less worry about concurrency.

[1] http://highscalability.com/blog/2016/1/25/design-of-a-modern...

[2] http://web.cse.ohio-state.edu/hpcs/WWW/HTML/publications/pap...

NovaX··on Caching Beyond RAM: The Case for NVMe
I wrote a simulator and link to the traces. One unfortunate aspect is they did not provide a real comparison with LIRS, except in a table that includes an incorrect number. It comes off a little biased since LIRS outperforms ARC in most of their own traces.

https://github.com/ben-manes/caffeine/wiki/Simulator

NovaX··on Caching Beyond RAM: The Case for NVMe
Thankfully there are better algorithms that are not encumbered. ARC is fairly memory hungry and requires a large cache to be effective. It is not as scan resistant and does not capture frequency as well as many believe. LIRS or TinyLFU based policies are what new implementations should be based on.

https://github.com/ben-manes/caffeine/wiki/Efficiency

NovaX··on Truth First, or Why You Should Mostly Implement Database-First Designs
I don't think anyone advocates for generating based on a production database connection. I can only tell you how I've used jOOQ, using the same general setup for 6 years despite job changes. I wrote the original Gradle plugins for jOOQ and Flyway, the latter being a migration tool. These were then contributed back and I use the official variants now.

I write SQL migrations for H2, an in-memory db. This is used by the build tool to code generate prior to compilation. It is also used by unit tests, when the Postgres queries can be emulated (majority of the time). The Postgres migrations are written separately, with similar code but minor dialect differences. That might be less duplicated (but trivial) work if I used LiquiBase instead of Flyway. When the application starts the migrations are run. Someday I might drop H2 for a containerized Postgres, especially if they add in-memory support for faster testing. This would avoid some codegen differences, like jsonb or (soon to be released) enum support.

The work setting things up wasn't too bad. It was more when I originally started, since I had to build part of the ecosystem. But repeating the process elsewhere has been straightforward. It definitely pays off and is a joy to use day to day.

NovaX··on [dead]
I don't believe Trump deserves the Peace Prize, but many of its winners have done despicable things. It seems

Turn away Jewish refugees from Nazi death camps - win a peace prize. (Cordell Hull)

Repeatedly refuse peace, call for the destruction of Israel and slaughter of all Jews - win a peace prize (Yasser Arafat)

Promote a war that appeared to have served no purpose than cause anguish - win a peace prize (Henry Kissinger)

I think we need two prizes, one for those who contributed and one for those who we hope might learn basic ethics.

NovaX··on LHD: Improving Cache Hit Rate by Maximizing Hit Density [pdf]
Can you provide examples of closed-source / unnamed algorithms? Most of the advanced algorithms, like ARC, can only provide a modest improvement to LRU. LIRS does quite well, but is complex and usually poorly implemented. The newest generation of systems have begun adopting TinyLFU, with research focusing on making it adaptive. I would be interested in understanding the unpublished work (you can reach out privately if desired: ben.manes @ gmail).
NovaX··on Why I Moved Back from Gradle to Maven
I would have used Bazel (or Buck) at Addepar instead of Gradle had either been released (I really loved Blaze). The transition from IDE-only to a standard process was painful, as you remember.

I'd like to use Bazel (or even Buck) these days, but no bandwidth given other projects. I always felt modules were a mistake compared to a single large tree, which Bazel does elegantly. Sadly my familiarity with Gradle is much more than Blaze these days, and I'm too exhausted to do it just for fun. I think there is also a bit of Stockholm syndrome at play, since I convinced myself it was not too bad back at Addepar.

NovaX··on History of Spring Framework and Spring Boot
I should have mentioned that the API flaws breaking with Bloch's advice can be really nasty. I bundled up a bunch of items under that, which might mundane if you don't know of them. Some of the wonderful design choices are,

- Cache.iterator() may return a null entry, despite null key or value is disallowed. This occurs when the cursor points to an entry that no longer exists - hasNext() was true, but the entry disappeared due to removal/eviction/expiration. This is a performance optimization to avoid underlying I/O that leaks to users and violates their assumptions.

- Cache.putAll(map) first invokes CacheWriter.writeAll(list<entry>) prior to storing them. The writer uses an in/out parameter so that if an exception is thrown, the user must remove the successfully written entries first. Then cache must store those successfully written entries before propagating the exception to the caller.

- An amusing bug was that their custom future could not be consumed by multiple threads. It used notify(), not notifyAll(), so only one waiter would wake up. The concerning part was that this existed for a few years due to lack interest in releasing a new version.

NovaX··on History of Spring Framework and Spring Boot
The API, though I am not a fan of the RI for one reason. It is the first JSR RI that I am aware of where the authors advice against using, it is unsupported, and intentionally low quality. All other RIs that I know of were production grade quality to ensure it met user's needs. This one was to pass a specification requirement.

- The API violates many of Josh Bloch's guidances [1], including items in Effective Java.

- Many of its features are incompatible with itself. For example, the AOP support cannot be used with loaders, writers, or listeners.

- The AOP support does not support atomic memoization (dog piling [2]) and can crash if initializing the same cache concurrently.

- The JavaDoc dictates implementation details that matched the lead's product, rather than being open ended to allow alternative algorithmic choices.

- The JSR defines an configuration API that is not suitable for production use and promotes the silent usage of unbounded anonymous caches.

- The interfaces do not attempt to feel natural alongside the Java's Collection Framework nor the direction that the language has been heading. It redefines terms, like "access", in an incompatible way.

- Despite numerous delays in being accepted, the JSR was not modified to take advantage of significant changes in the language (java.time, lambdas, CompletableFuture, etc).

- The TCK is not robust, does not check concurrency, and bugs in it languished.

- Most importantly, writing code using JCache is not enjoyable and it is often not flexible enough to solve most needs.

There are likely many more, but that is a short list that comes to mind in a few minutes.

[1] https://www.infoq.com/articles/API-Design-Joshua-Bloch

[2] https://en.wikipedia.org/wiki/Cache_stampede

NovaX··on History of Spring Framework and Spring Boot
4. JCache (jsr107) is really awful. I don't think you want one.
NovaX··on Linux Torvads calls Intel patches 'complete and utter garbage'
He is very defensive and dismissive of his own mistakes. Linus suggested the removal of a memory barrier and subsequently broke locking on kernels for most of 2014. The fix did not get into most distributions until 2016, since it was kept quiet and not communicated. The result is that deadlocks seemingly occur at random, causing complaints from users of my libraries. It has wasted many hours of my time, and surely more for others. However the more experience you have, typically the more forgiving you are to mistakes as we all make them. Linus lacks that and acts childish, making it very difficult to respect someone who disrespects everyone else.
NovaX··on Ask HN: What was the best CS paper you read in 2017?
Interestingly this work has nearly identical algorithm to a previous paper. Good ideas are rediscovered.

Counting with TinyTable: Every Bit Counts! http://www.cs.technion.ac.il/users/wwwb/cgi-bin/tr-get.cgi/2...

NovaX··on Data Classes for Java
Nope, we just add @AutoBuilder

https://github.com/h908714124/auto-builder

NovaX··on Ask HN: What tech that's right around the corner are you most excited about?
We're building a similar at product at https://loaddocs.co and hiring, if you are interested.
NovaX··on Ask HN: What tech that's right around the corner are you most excited about?
What's the app?
NovaX··on Google accuses Uber of creating a fake company in order to steal its tech
Google actively assisted in selling illegal pharmaceuticals due to the lucrative ad revenue. It was coordinated and driven by the chief executives.
NovaX··on Cache eviction: when are randomized algorithms better than LRU? (2014)
I used hill climbing to tune TinyLFU. A larger admission window improves recency-biased workloads, while a smaller improves for frequency. By sampling the hit rate and making an initial adjustment, it can determine when to change directions and hone in on the best configuration. Other than a small warm-up penalty, it quickly optimizes itself.
NovaX··on Cache eviction: when are randomized algorithms better than LRU? (2014)
ARC has modest scan resistance due to a limited history. Like LRU, it suffers from cache pollution. See the glimpse trace for this workload type [1].

LIRS and TinyLFU make better use of their history to filter out low value items early. The fundamental difference in their designs is starting point: LIRS starts from LRU and infers recency, whereas TinyLFU starts from frequency and infers recency. There is work on an Adaptive TinyLFU that mimics LRU by using an admission window, sampling the hit rate, and using hill climbing to adjust the region sizes to best fit the workload.

[1] https://github.com/ben-manes/caffeine/wiki/Efficiency#glimps...

NovaX··on Taking a Look at CLR Thread Pool 'Thread Injection' Algorithm
I have a prototype to optimize a caching policy using a hill climber. It favors recency or frequency by adapting the partition between the two regions. It works really well in early experiments.

https://docs.google.com/presentation/d/1NlDxyXsUG1qlVHMl4vsU...

NovaX··on [dead]
I found reading the Talmud and Pirkei Avot to be helpful in interpreting the (Jewish) Bible. I also read a lot of history books which made it easier to understand the meaning behind what the authors were trying to convey, the challenges, and the culture. (As a Jew, I found the NT to make absolutely no sense and frustratingly illogical / inconsistent).
NovaX··on Random notes on improving the Redis LRU algorithm
I also ran an exhaustive analysis and had similar conclusion. I found that TinyLFU (mentioned in the article) matches or beats LIRS in database and search workloads. It is also much simpler to implement and is more memory efficient.

https://github.com/ben-manes/caffeine/wiki/Efficiency

The W-TinyLFU variant better handles recency by using an admission window so that new entries are not immediately discard. This resolves antirez's concern and improves performance in sparse burst workloads. In a way its similar to the HIR/LIR regions (window/main) using TinyLFU as a frequency filter.

Happy to work with you, antirez, and others to simulate more policies and add trace files.

NovaX··on Synchronization primitives in the Linux kernel, Part 4
There's no either because caches are queues in disguise

http://highscalability.com/blog/2016/1/25/design-of-a-modern...

NovaX··on Ask HN: Cache Hit Ratio
This is sometimes referred to as the "admission policy" of a cache and can have a large effect at increasing the hit ratio.

Most policies try to minimize the impact of low frequency items. For example SLRU uses a small probation space that promotes to a protected region if re-accessed. This allows quickly discarding low frequent items. ARC uses an adaptive version of this idea.

LIRS uses a very small region (HIR) that promotes to a large region (LIR) based in the inter-reference period. This allows it to discard many new arrivals by using a large number of ghosts to retain usage history.

TinyLFU is a probabilistic LFU filter that decides whether to admit by comparing the frequency of the arrival to the victim. This is often near optimal, but can suffer due to sparse bursts causing consecutive misses. An admission window (W-TinyLFU) resolves this problem, usually with a 1% window.

I have a simulator and traces that you could easily add onto to experiment. [1]

I recommend reading the TinyLFU paper as I think it is the best match to your ideas. [2]

[1] https://github.com/ben-manes/caffeine [2] http://arxiv.org/pdf/1512.00727.pdf

NovaX··on Apache Arrow: A new open source in-memory columnar data format
"Modern CPUs are designed to exploit data-level parallelism via vectorized operations and SIMD instructions. Arrow facilitates such processing."

How will Arrow use vectorized instructions on the JVM? That seems to be only available to the JIT and JNI, which is a frustrating limitation.

← PreviousPage 5 of 6Next →