HNHacker News
TopNewBestAskShowJobs

trengrj

1,241 karma · joined August 14, 2013

applied research @ weaviate https://x.com/trengrj
submissionscomments
trengrj··on AMD is sending Ryzen/Radeon care packages to game developers
Does anyone have any good info on Ryzen ECC RAM support? I’ve read a bunch of conflicting information and that some motherboard vendors have been misleading.
trengrj··on Ask HN: Is there a place to learn Hadoop/Hive/Big data analysis through cloud?
As mentioned by maa5444, Databricks community edition is a good way to learn Spark as you get a free 6GB node. If you want to focus more on Cloudera/Hortonworks, I would just spin up VMs in AWS or your preferred cloud. The costs of running test nodes (say 32/64 GB ram 100GB disks) are not super high if your are economical with shutting them down when not in use.
trengrj··on Hadoop 3: Comparison with Hadoop 2 and Spark
This article (like a lot of others) confuses Spark and Hadoop. To clarify:

Spark is a data processing engine. Spark works with many persistent storage layers but provides none itself. So Spark can run on object storage (S3 etc), additionally it can run on HDFS (the Hadoop file system).

The right layer to compare Spark with is MapReduce or to compare its SQL engine vs other SQL engines that run on Hadoop (Impala, Hive LLAP, Presto).

YARN is used for resource scheduling in the Hadoop world but Spark has other options like its internal scheduler, Mesos, and now Kubernetes.

When people say Hadoop, the technical meaning is often HDFS + Hive + YARN + MapReduce (for legacy) + implicitly Spark on Hadoop. All Hadoop distributions support Spark.

So the comment Spark is faster that Hadoop is like saying Postgres is faster than Google Cloud.

trengrj··on The Crystal Programming Language
I found it a bit weird too as introducing a cyclic binary dependency is definitely not best practice for a standard programming project. However the C compilers do the same thing and so you already have these types of cyclic binary dependencies and it is unfair to force all languages to be written in C.
trengrj··on Wizard for Mac – new kind of statistics program
I'd recommend some screenshots or the video before you get to pricing.

Some of the graphics need to be improved. It would also be great to see why this is better than tradition BI tools (Tableau etc) and what your unique value proposition is.

trengrj··on Sway 1.0-alpha.1 released
I used to use tiling window managers like bspwm and dwm but now just use Fedora and GNOME. I am worried that with the move to Wayland that all the great window managers will break.

It is good to see Sway is approaching 1.0 and that they are also producing a library https://github.com/swaywm/wlroots for building tiling window managers as I believe currently something like the 700 line monsterwm can't exist in the Wayland world due to the sheer amount of boilerplate and auxiliary code required.

trengrj··on Google Bans Crypto-Currency Mining Extensions from Chrome Store
I see the fact we can now easily trade cpu power for currency to have many more unexpected effects.
trengrj··on Notice to stakeholders: withdrawal of the UK and EU rules on .eu domain names
I dislike the idea that domain names can be geographically restricted. Additionally this .eu cancellation seems a wanton and bureaucratic attack on the U.K. given Brexit has only recently been triggered.
trengrj··on CockroachDB 2.0 Performance Makes Significant Strides
Looking at the TPC-C page all the benchmarks seem quite old and only reflect commercial databases. Do you have any recent TPC-C benchmarks for OLTP databases such as Postgres, MySQL, and Cassandra so I can compare with CockroachDB?
trengrj··on Quiver – The Programmer's Notebook
I use Typora https://typora.io/ which has most of the functionality including Markdown, Code blocks with syntax highlighting, LateX segments, and is more importantly free and cross platform.

It has recently added file list preview and tree view like functionality. Only missing on tags which are tricky because they break its pure markdown flat file philosophy.

trengrj··on Cryptocurrencies: Last Week Tonight with John Oliver (HBO)
This is geoblocked in Australia. Does anyone have an alternative source?
trengrj··on Ask HN: How to self-learn math?
I would select books based on your interest. I find the Dover publications good because they are both cheap and slightly older, this means they are less focused on undergraduate monetisation (version hopping, not supply answers to problems, glossy print), and more focused on proofs and algorithms. You can see them here http://store.doverpublications.com/by-subject-mathematics.ht.... I particularly liked Probability: A concise course, and Number Theory by George Andrews.

Amazon used to have a great number of graduate preparation book lists which always included books such as Rudin's Principles of Mathematics Analysis, and Halmos' Finite Dimensional Vector Spaces. These classic maths books are brilliant but usually easier to understand if you already have some experience with the material.

Final advice is to find a study partner as it can be hard to track how you are going and keep motivated, especially without the instant feedback loop you get with programming.

trengrj··on Anatomy of a Ceph meltdown
Obviously as others have mentioned there were a lot of mistakes here around engineering best practises: using a bleeding edge source based distribution for critical storage infrastructure, allowing drift of package versions, and not testing releases.

I've done some work in Ceph and in my opinion it is too ambitious: block, object, and file system storage, custom raw partition format (Bluestore), along with a somewhat legacy C/C++ codebase makes the system fairly fragile. I would only recommend it if you "know what you are doing" or are willing to get some help from Red Hat.

The other comment I'd make is that 12TB unreplicated (assuming they are using 3x replication for Ceph) is actually not a huge amount of data and in my opinion a ZFS setup would be cheaper and more stable. It is not too challenging to do a HA ZFS setup, and also ZFS's many mirroring, scrubing, and checksuming abilities makes it very resilient. Copy on write and ZFS snapshots are also a great features that can save on disk usage.

I feel the sweet spot for Ceph is probably 50TB - 1PB where you need both block and object storage and are unwilling to use cloud solutions. Lower than 50TB and the overhead and risk of managing Ceph makes it less practical than traditional solutions. From the other direction CERN did testing up to 30PB with Ceph but they had to make significant code changes and had Ceph committers in their team. To compare Hadoop is running in clusters of up to 600PB (but you will be accessing the data very differently).

[1] https://cds.cern.ch/record/2015206/files/CephScaleTestMarch2...

trengrj··on Impala: Scalable Distributed Deep Reinforcement Learning
I'm starting to this we need namespaces for open source projects so we could refer to deep-learning:impala and database:impala. The java reverse domain name notation does handle this to a degree but I feel something could be done in this area to make the naming issue easier.
trengrj··on People who are less willing to apologize also tend to be less self-compassionate
In other news people who are less willing to smile also tend to be less friendly.
trengrj··on CloudFlair: Bypassing CloudFlare using Internet-wide scan data
It would be good if AWS et al had an easy way to manage third party ip ranges for security groups. When I was deploying a site we came across this issue and so whitelisted Cloudflares ip ranges but it made me uneasy because what happens if a new IP address is added? How do we manage this list and what sort of notice is provided?
trengrj··on Western Digital My Cloud drives have a built-in backdoor
For reasons like this I built my own NAS from scratch using a server and ZFS (admittedly using Western Digital red drives). I don’t believe consumer NAS vendors have the capability to properly patch and secure web interfaces and I’d extend this to Synology and QNAP as well.

Additionally NFS and SMB are very complex protocols and difficult to setup correctly. NFS without Kerberos provides only very basic whitelisting security. Adding Kerberos means you need active directory or something like FreeIPA and a scary amount of configuration (DNS server requirements, client side Kerberos config etc). I went down the whole Kerberos route and at the end decided mounting via ssh was far simpler and probably more secure. I would only trust a very technically competent vendor to do this well and don’t know if one exists.

trengrj··on Amazon Macie: A machine learning service to discover and protect sensitive data
How does this compare to Google’s Data Loss Prevention?
trengrj··on Stacks-cli: Analyze a website’s stack from the terminal
I just use BuiltWith.com for this.
trengrj··on Ask HN: How’d you go from a noodler to writing large well structured programs?
Have a read of Clean Code and the Pragmatic Programmer.

I’ve seen this happen before in teams that code all day but don’t see themselves as developers. You need a instigate a cultural change at work where it becomes unacceptable to not apply modern development practices just because the team sees themselves as “quants” first.

trengrj··on They’re Talking Behind My Back: Remote Workers Feel Unsupported
Thanks your article is well written and showed me a different approach to management.
trengrj··on Speech Recognition Is Not Solved
We recently did a comparative analysis of cloud speech-to-text providers for a project. We looked at:

1. Google Cloud Speech API

2. Microsoft Bing Speech API

3. IBM Watson Speech to Text

The ranking was as listed above but we had real challenges working with call-center audio recordings. The quality was less than idea but still very clear. We saw a huge reduction in accuracy compared to in-browser testing. Additionally Australian English is particularly not solved.

Because Google's API isn't currently doing speaker-detection, we looked at using Watson's speaker-detection as a secondary step but found it too complex and error prone. There is definitely room for a startup in this area and it also needs continued investment from the bigger cloud providers.

trengrj··on Google Data Studio
Usually a notebook style interface with Juypter / Zeppelin is best for this provided the user has enough capability.
trengrj··on Org-Mode Is One of the Most Reasonable Markup Languages to Use for Text
Yes!
trengrj··on Org-Mode Is One of the Most Reasonable Markup Languages to Use for Text
I love the concept of using lightweight markup text but a lot of the comments in the article also apply to Markdown. Personally I've switched to using Markdown for everything possible and have started using Typora as my rich text editor as it provides the first fluid WYSIWYG markdown interface I have seen with easy export to PDF / word / HTML.
trengrj··on How We're Building a Business to Last
I'm actually quite happy with the name. For me - cockroaches are bugs that are distributed and survive in harsh conditions - something you want out of a database.
trengrj··on Lenovo Accidentally Leaks an Image of Its ‘Retro Thinkpad’
The gpdpocket is pretty close if you are looking for a trackpad less design (though too small for daily use). If you can afford 100,000k for a mould they could be you best bet for doing something custom.
trengrj··on Vagrant 2.0
Yeah agreed. I just wasted 2 hours this week compacting my qcow2 docker image for my mac (by filling it with zeros). It was totally unintuitive that deleting images didn't free disk space.
trengrj··on Facebook recruiting and Unix systems
He is actually incorrect in this case as Mac OS is certified UNIX so is a UNIX and not just UNIX-like [*]. If the recruiter was being clever and wanted the applicant to understand the distinction or answer with the magic OS X words I'd be impressed.

https://unix.stackexchange.com/questions/1489/is-mac-os-x-un...

trengrj··on Run Kubernetes on top of DC/OS
Excellent. Thanks!
← PreviousPage 2 of 7Next →