HNHacker News
TopNewBestAskShowJobs

manish_gill

1,516 karma · joined October 18, 2011

Software

https://parallelthoughts.xyz

https://twitter.com/mgill25

submissionscomments
manish_gill··on Ask HN: Am I the longest-serving programmer – 57 years and counting?
+1 I'm deeply interested. Been thinking a lot about doing a Masters in AI/ML but I'd rather not stop working. Been considering part time masters programs but finding them is tricky.
manish_gill··on Racket is 25
I actually finished that course when it first came out (and was a single course as opposed to a 3 part breakdown) way back, like 4-5 years ago.

Thanks all for the suggestions! :)

manish_gill··on Racket is 25
I've picked up Racket recently after a long while and I'm enjoying it.

Does anyone have recommendations of how to learn to think recursively? Sure, using the language more and more does it, but what about a more general recursion as a problem solving paradigm pattern? I'm enjoying doing recursive code exercises and got into Backtracking problems. Creating things like Sudoku solvers, Knight's Tour grid etc was quite fun. I still can't develop recurrences easily though.

Any books/exercises that people here found helpful?

manish_gill··on Ask HN: What is your blog and why should I read it?
https://parallelthoughts.xyz Only a couple posts so far, one that made HN and Reddit front page. I mostly write about data engineering, and have some systems posts in the pipeline - experiments with Rust, Performance Analysis, debugging stories etc.
manish_gill··on Ask HN: What are you learning?
Graph Algorithms and Distributed Systems
manish_gill··on Ask HN: What projects are you working on now?
My first thought was also Clickhouse and the problems with schema for dynamic log content.

Also, if this is open source, I will definitely be interested in checking it out/contributing etc.

manish_gill··on Using Anki to remember what you read
Looks great! I have a few questions.

- I have a huge Zettelkasten note archive (I manage it through Sublime-Zk, which is getting a bit slow to operate it). Can I somehow directly import all my Zettel notes?

- I tried to import my Anki cards, but it seems like while I could go through them, I couldn't assign them to any particular deck/reclassify them. They show up as part of a "Not Found" deck and I don't see any option/drag-and-drop method to move them anywhere else.

I'll definitely be giving Mochi a try. Thanks a lot for creating it!

manish_gill··on Ask HN: How do we stop the polarization/toxicicity filling the web?
The problem is when people with such beliefs acquire position of power or influence, which sadly has already happened.
manish_gill··on A Sad Day for Rust
This is why I wrote "forked". They do have the right to not accept PRs for their own project, no? Sure, someone can be a bad steward, or maybe just opinionated. Doesn't give anyone the right to be mean to the point of making them quit their own project.
manish_gill··on A Sad Day for Rust
> No one expects you to write perfect code, but we do expect you to fix flaws when you learn about them.

It's not like he was getting paid to work on this, was it? And people do have a life beyond open source. People could have forked and worked on the issues themselves, but that's asking too much. Why do the hard work when you can just write a comment/tweet blaming someone else, right?

Your comment is precisely what entitlement looks like.

manish_gill··on Ask HN: Who wants to be hired? (October 2019)
Location: Berlin

Remote: Yes

Willing to relocate: Yes

Technologies: Python, SQL, Java, Go, Lua, Rust (learning), PostgreSQL, Kafka, BigQuery, Spanner, Beam/Dataflow

Résumé/CV: https://parallelthoughts.xyz/resume.pdf

Email: contact@manishgill.com

I'm a Data Engineer, who has built teams from scratch, and delivered end to end customer-centric data platforms. Looking for an interesting challenge in a high scale environment.

manish_gill··on Ask HN: Are you ok?
Don't know. I moved to Europe a month ago, leaving friends family and girlfriend behind, only to find out that the company might be facing a tough time ahead - CTO has left, primary revenue sources are expected to take a downfall and investor confidence is low.

I was actually hoping to do some sort of part-time Data Science/ML Masters along with my work, but looks like I'll have to get back on the interview grind pretty soon. Not looking forward to getting back on the Interview grind - take home assignments which last a week or so, or intense algorithms training. Next company will definitely be a stable/big tech one. But for that, I'll need to leetcode like crazy.

Unsure of my future.

manish_gill··on On SQS
I've successfully been using Kafka in production with a tiny percentage of packets ~1Mb. We utilize gzip compression with chunking on some of those messages (system is legacy, no fancy compressed message format in use). IIRC, only had to modify settings at the broker level and it works perfectly fine.
manish_gill··on A Tale of Query Optimization
Thanks, I'll take a look at this approach.
manish_gill··on A Tale of Query Optimization
BETWEEN is soemthing that is on the "Don't do this" page of Postgres. :)

https://wiki.postgresql.org/wiki/Don%27t_Do_This#Don.27t_use...

the ::text[] addition had no impact on perf. I actually constructed this post from my notes at the time so might missed have a few things here and there.

The bit about query planner recognizing things is exactly what the challenge is to be honest. Like I said at the end, the good mental model of SQL helps a lot, and mine is decent-ish I hope but can still learn a lot. :)

manish_gill··on A Tale of Query Optimization
This is something I didn't consider. Will definitely try it out. Thanks!
manish_gill··on Emacs 26.2 Released
Emacs is the one environment that I've been a fan of from afar. I recognize the brilliance and power of it, but a few things have always kept me away from using it on the Mac. And I've put in serious attempts, including going emacs-only, then a custom evil mode, then spacemacs -

- The server mode compatibility with the way Mac applications are set up. I'd ideally want an emacs server running in the background, and a client app open/close seamlessly whenever I'd like. What actually ends up happening is that I close the client windows but the app never "quits". So it's a zombie icon always hanging around my cmd+tab list.

- "Windows" (or Frames?) have generally not worked seamlessly for me. I've to always put in hacks in my config to ensure the emacs windows work like other apps do for mac. They're either too small and then size up when config is applied (which has some latency so it's noticeable when it happens), or they don't work at all.

- Font rendering. There is some minute rendering difference in the way the same fonts that renders on Vim/iTerm/Macvim/Sublime/VSCode/IntelliJ, but it's the worse by far on my emacs. :(

- All of these complaints are for the GUI mode tbh, which I prefer to use since I like to keep my terminal separate from my editor. Terminal-only emacs is something I tried a few years ago and I remember getting annoyed by keybinding issues etc.

Maybe all of the above are old issues that are no longer present, or maybe I was doing things the wrong way. It just never felt like it was a seamlessly integrated application on the Mac. I hope that has changed or will change. :)

manish_gill··on Kong Raises $43M in Series C Funding
We've been using openresty in production at vwo.com at our edge nodes for quite a few years. It's really fantastic, allowing you to write custom logic and embed it within nginx itself, making things super fast.

The only real disadvantage is the lack of a good library ecosystem within lua and openresty. Over the years, as we moved some more logic to edge because $reasons, we ended up rolling our own libs quite a few times. Documentation can be made much easier as well.

That, and the occasional limitations of Lua as a language.

Overall though, quite an awesome project.

manish_gill··on How to pass a programming interview (2016)
> Mentioning other offers in an interview heavily biases the interviewer in your favor.

Is this really true? Also, how do you just mention offers? And will the programmer interviewing me really care? Should this only be done at the HR stage?

Like, what's a good way to let the other person know that they're not the only one.

manish_gill··on SQL: One of the most valuable skills
> On the other end of the spectrum are the influx of "data engineers" with basic to intermediate knowledge of HDFS, streaming data or various other NoSQL technologies. They often know less about raw SQL than even junior engineers because SQL is below their high-power big data tools.

I always ask a basic SQL question (involving sorting/grouping and one simple join) in my interviews for backend and/or data. IMO if you are calling yourself a Data Engineer and do not know SQL, then you haven't really worked with data, you've mostly just worked on data infrastructure.

manish_gill··on SQL: One of the most valuable skills
SQL is pretty great. I've been doing advance SQL in Postgres and BigQuery for quite a while and it's one of the most satisfying experiences.

Once you wrap your head around basics such as GROUP BYs, JOINs, CASE statements etc, you move on to advance concepts such as Window functions and there suddenly a new world of possibilities opens up to you for analytics! I've dabbled a bit in PL/pgSQL, but the syntax is way too arcane.

SQL can also be counter-intuitive sometimes. I rewrote a particular PG query and reduced time from ~30minutes down to 2 seconds by adding a subquery. :)

And no, ORMs can't really do what SQL can do in an equivalent manner.

manish_gill··on SQL Is No Excuse to Avoid DevOps
> Anybody doing this on a production database without a strictly controlled (and tested!) break-fix should be fired on the spot.

I agree with almost everything you said except this line. Usually, people with production access are (or should be) sensible enough not to touch the prod db with anything untested that might mutate things.

If they're not sensible enough for that, they shouldn't have had access in the first place, and instead of placing blame and firing people, I'd start thinking about /why/ that happened. Points to a more fundamental flaw in the Engineering. Firing won't fix anything.

Also note that the statement in question talks about an old methodology.

As for the article, good stuff. I enjoyed reading it.

manish_gill··on Ask HN: How is software development different in China and India?
Depends entirely on your company. The most important distinction I've come across is whether someone is from a Service-based company or a Product company. Unfortunately, the service industry leaves a lot to be desired in terms of quality. I work for vwo.com (a Product company) and I can speak a bit about my workplace:

Programming Languages: We use heavy Javascript for our core libraries, VueJS and Angular for frontend (different projects), PHP/MySQL backend (legacy) with our own home-baked framework. We also have our own edge nodes with lots of OpenResty/Lua, and our data pipelines are in Python, Java and Scala.

Language: English predominantly, but discussions and meetings can be a mix of English and Hindi (we're based out of North India). I switch to English exclusively if there's a teammate who doesn't understand Hindi very well.

"Is Agile used": I think the question is broadly about engineering practices and yes, we do use Agile/Jira, doing 2-week or more Sprints in cross functional teams. It's not perfect and everyone has their own complaints with this model, but oh well. We do heavy code reviews and have a dedicated QA team - for both automated and manual testing, extensively E2E.

Culturally, we're pretty open and transparent. Everyone /broadly/ knows what everyone else is working on. It's not startup culture, but as far as mid-sized companies go, it's pretty good. Not a whole lot of politics, a "get the job done" attitude. We occasionally open source some projects via Github. Work from home/Vacation policies are decent. I'm currently on Christmas holidays actually ^_^

manish_gill··on Ask HN: What technologies did you learn in 2018?
Databases and Streaming Systems. Apache Beam/Cloud Dataflow. GCP Data Stack: BigQuery, Spanner, PubSub. Moar Apache Kafka.

LOTS of SQL. Writing sophisticated queries, building a mental model of what a DB will try to do, doing perf evaluation by looking at query plans, Analytic/Window functions.

DB optimizations. Especially BigQuery and Postgres.

Non-trivial Java 8.

Holidays are for Rust, advanced Streaming systems, and Docker/K8 :-)

Will also be looking to get deep into the Python AI/ML ecosystem in 2019. I'm fairly adept in Python and thankfully the ecosystem for Data Science/ML is quite mature.

manish_gill··on Why Not to Build a Time-Series Database
As I was reading through the post I kept wondering why they weren't using some warehousing technique for older data - either dump it to S3 or better yet, Google BigQuery, which is amazingly fast at that scale. They only did it after doing lots of fire-fighting and per-tenant clusters.

Clickhouse would also be a good option for doing aggregating queries that TSDBs are mostly used for.

One of my wishlist items in the data space is a Managed Clickhouse offering. :-)

manish_gill··on Uber’s Big Data Platform: 100+ Petabytes with Minute Latency
Good post. The Snapshot-based approach during ingestion time was the part where I couldn't figure out why it was considered a good decision during implementation?

I've experimented with Parquet data on S3 for a work POC, and the latency to fetch the data/create tables/run the Spark-SQL query (running on EMR cluster) was quite noticeable. I was advised that EMR-FS would make it run quicker, but never got around to playing with that. But I guess the creating of in-memory tables using raw data snapshots would still remain true? Or maybe I missed something.

Also, I take it if 24 hrs is the latency requirements for ingestion to availability of this data, obviously this isn't the data platform that is powering the real time booking/sharing of Uber rides. I'd be curious to see what is the data pipeline that powers that for Uber.

manish_gill··on Ask HN: What are some of the best documentaries you've seen?
A few I've seen recently:

1. Vietnam series by Ken Burns 2. Icarus 3. Barca Dreams

manish_gill··on Fennel – Lisp in Lua
I find your point about Lua's ecosystem to be interesting. I have been working with Nginx/Openresty for quite a while now and I've found the Lua library ecosystem to be quite a bit lacking.

I like Lua quite a bit and it is of course, extremely fast, but I wish there were good "batteries" along with other stuff that I've come to expect from Python/Ruby et all. Of course, I've coded up some of those libraries myself as well. :)

Just wondering what people's general thoughts on this are.

manish_gill··on Dark Motives and Elective Use of Brainteaser Interview Questions
I took an interview today and was asked a simple problem which I immediately answered with linear time complexity. The interviewer then went on to say "no I'm looking for a specific answer which involves a trick".

Literally said those words.

Eventually figured it out but I came away with a bad impression of the company. Not the kind of stuff I judge people on.

manish_gill··on India Pushes Back Against Tech ‘Colonization’ by Internet Giants
"Colonization" is a heavily loaded word and it's frankly, immature from NYT to use it. Data Protection Regulations (in the same vein as GDPR) are a good thing for consumers.

As someone who is concerned about user and data privacy, I like these developments. What's anti-colonial about it other than just a catchy headline?

← PreviousPage 4 of 15Next →