HNHacker News
TopNewBestAskShowJobs

artwr

163 karma · joined April 7, 2017

Data Engineer

https://github.com/artwr/ https://twitter.com/awiedmer

submissionscomments
artwr··on Science needs more research software engineers
I will second this. The national labs are definitely some of the places who know what to do with research software engineers and treat them right for the most part. The Computer Systems Engineers and Software Engineers I had the chance to meet at LBNL had a decent amount of autonomy and were very good.
artwr··on Lessons learned from running Apache Airflow at scale
`your_task.override(task_id="your_generated_name")` not working for you?
artwr··on Lessons learned from running Apache Airflow at scale
At Airbnb, we were using SubDAGs to try to manage large number of tasks in a single DAG. This allowed organizing tasks and drilling down into failures more easily but came with its own challenges.

In more recent versions of Airflow, TaskGroups (https://airflow.apache.org/docs/apache-airflow/stable/concep..., https://www.astronomer.io/guides/task-groups/ ) were made to help this a little bit. Hopefully that helps a bit.

At ~1k nodes in the graph introspection becomes hard anyway, as others have suggested, breaking it down if possible might be a good idea.

artwr··on Lessons learned from running Apache Airflow at scale
Can I ask more about your use case that you could not find an answer for?
artwr··on Pixel prevented me from calling 911
Thank you for reminding me of the IT crowd. Now I have the music in my head
artwr··on What's the Future of IDEs?
I would tend to agree for most languages I have touched (Python, C, ...) but I have mostly worked in medium codebases.

Do people think it is still the case for large code bases and Java? I find the ability to quickly jump to definition in the IDE a huge help when on boarding into a new codebase.

artwr··on France moves to shield its book industry from Amazon
Was it Charles Petzold's Code? If so that's a wonderful book.
artwr··on Facebook bans, sends C&D letter to developer of Unfollow Everything extension
#latexforlife
artwr··on AWS Identity service handles 400M API calls every second
A lot of the calls in the organizations I have worked for use assume role and are usually associated with temporary credentials. Those are often shortish lived and not static
artwr··on Berlin rent cap overturned by Germany's top court
Just curious, would you happen to have a reference for Paris handy?
artwr··on Show HN: Simplenetes – I replaced Kubernetes with 17k lines of shell script
Asking to learn, but if you were to start a Java application in the following two scenarios: 1) Monolith with some CRUD, API, persistence to a DB + search. 2) Scalable backend with an API, persistence to a DB + some caching

what would you use these days?

artwr··on Apache Drill's Future
The original SparkSQL was pretty much modelled after the Hive flavour of SQL, down to the available udfs. The compatibility was never complete and has somewhat diverged again with respective releases of the frameworks, but for the most part, Hive was the big data framework to beat at the time (2015-2016), and not everyone wanted to write Scala.

I think that now, maintaining that compatibility is less of a need for Spark and Hive has introduced a lot of goodies in the meantime, so there might not be a need for the SQL flavors to be in lockstep anymore.

artwr··on Medium-hard SQL interview questions
Two ways to address this: - Join to a date dimension and coalesce nulls to 0 - use RANGE BETWEEN (only available in some DBs) on a date_difference column
artwr··on Taiwan says WHO failed to act on coronavirus transmission warning
1. https://twitter.com/WHO/status/1217043229427761152

2. https://www.reuters.com/article/us-china-health-who-idUSKBN1...

3. https://www.cnn.com/2020/02/04/health/who-coronavirus-not-pa...

Can't comment on 4.

5. it's a little bit complicated as I believe they are just not recommending masks unless you are sick or caring for the sick.

artwr··on How to Write a Book in Emacs (2015)
A former teacher of mine was working on a way to do this with pandoc and Rmarkdown. The result is called redoc: https://github.com/noamross/redoc It's still in beta though.
artwr··on A Quick and Easy Guide to Tmux (2015)
Not the parent poster but for some financial services companies (bank, insurance), I have heard of development being done on a remote desktop, sometimes without direct ssh access.
artwr··on How shrinkflation is playing havoc with economists’ models
Cue the heavy cardboard rolls in the middle.
artwr··on The Heart of a Swimmer vs. the Heart of a Runner
True, but don't neglect leg day :)
artwr··on Lyft Crashes Below IPO Price
Thanks for the appropriate TL;DR
artwr··on I Was Kicked Out of the Society for Classical Studies Annual Meeting
It seems like the article was removed, but a copy is available here: http://archive.is/65K8o
artwr··on Myths in Machine Learning Research
Tensors are in effect a generalization of matrices in higher dimensions. A tensor of dimension 3 is a similar step up to the step from a linear array to a matrix. They arise in all sorts of places though fluids is where I met them first. Operations on tensors are also a bit more challenging than operations on matrices.
artwr··on The Risks of Water Insecurity
While the article mostly provides a few links, it's a topic that is worth looking at a little bit.

There is a lot to discover, from the fact that the water quality in the US as a whole is not great for a first world nation and the fact that some areas in the US where people actually live have water which is actually dangerous to consume (Flint).

Did you know that the several aqueducts in California carry naturally occurring asbestos and may be responsible for cancers (https://www.ncbi.nlm.nih.gov/pubmed/7395855) ?

artwr··on John Stockman’s Medical Bills Topped $1M – What Happened?
On the other end, medical degrees do not amount to ridiculous student loans in those countries either. Tuition is usually on the order of $10k/year.
artwr··on Ask HN: How do you negotiate a big raise in startup equity?
Without more info, it's a little tough to tell. A few parameters which would influence an answer: - How big is the startup? 10 employees? 50? 100? - Do you have an outside offer? - What makes you think you deserve a raise?

But otherwise, usual rules of negotiation: - establish the relationship with the decision maker, - What is a win win for you and the company? How is them giving you more equity going to put the company in a better position? - Early stage startups look for believers and an outside offer can be interpreted as a negative signal from that point of view. Step carefully.

artwr··on Quora pays thousands for posting questions
Or -site:quora.com if you are OK with pages with links to quora but you just do not want a result from that domain.
artwr··on Harvard Quietly Amasses California Vineyards and the Water Underneath
It really depends on how well the program/department is funded. Computer Science usually is. I was in a program (Environmental Engineering) where most masters degree students had to pay full price. Grants or graduate student positions wouldn't kick in until year 2. Even then, full tuition was never a guarantee. This was UC Berkeley.
artwr··on Ask HN: Is there a better way to document complex software architectures?
Ha-ha, the joke is on you, the systems I have inherited do not have tests.

Joke aside, tests are a great resource when they are present. The other technique I have used was to look at the system like a black box, and only look at the expected behavior at the boundary. Sometimes, it is the fastest way to clear old cruft.

artwr··on Ask HN: Best Command-Line Applications?
A few of the ones I use almost daily

- Pandoc a handy document format converter: https://pandoc.org/ - Ripgrep, a faster grep: https://github.com/BurntSushi/ripgrep - There are quite a few prompt-toolkit based CLI which have been really nice: * mycli (https://github.com/dbcli/mycli), pgcli (https://github.com/dbcli/pgcli) * ptpython/ptipython (https://github.com/prompt-toolkit/ptpython) - Youtube-dl: http://rg3.github.io/youtube-dl/ - fzf a fuzzy finder for the console: https://github.com/junegunn/fzf - jq for JSON manipulation: https://stedolan.github.io/jq/ - Miller for CSV/TSV: https://github.com/johnkerl/miller

I would imagine you have already scoured the "awesome" lists: https://github.com/agarrharr/awesome-cli-apps#terminal-utili...

artwr··on Ask HN: Anyone else find the new Gmail interface sluggish?
Thanks for the informative benchmark. I am (slowly) considering alternatives, are you also considering making the jump?
artwr··on Ask HN: Anyone else find the new Gmail interface sluggish?
Nvidia? Netflix?
← PreviousPage 2 of 4Next →