HNHacker News
TopNewBestAskShowJobs

ibains

226 karma · joined July 26, 2019

submissionscomments
ibains··on California population declines for first time in more than a century
We should be building Fabs in California, Cars in California, getting the smartest people in the world here, in solid numbers. Perhaps we’re in a global competition and we’ll not be the largest economy soon, perhaps we’re too busy with pronouns to notice.

People pattern-matched that the house prices always went up, assumed it was an axiom, and that drove us into a recession. Gavin Newsom will observe that California always grows and people have been talking of leaving the state for years, they never do - axiom. Basically, physics (cause-effect) stops mattering if you pattern match history.

Biggest threat to California is being taken over by the employee-bureaucracy class - that never has built anything. It is in their nature to pattern match so de effects of actions and treat them as axioms, and they miss the big picture, and they’re consistent as hell in it. And they vote very left.

ibains··on Ask HN: Who is hiring? (May 2021)
Prophecy.io - Building Low-Code Data Engineering product for Spark and Airflow.

Compilers, Parsers, Functional Programming, Database internals - you know one of these, or you’re a solid engineer who is interested in them.

US | Europe (San Francisco Bay Area preferred)

200-300k + generous equity. Reach out to raj@prophecy.io

ibains··on Ask HN: Who is hiring? (April 2021)
Prophecy.io - US/Europe - Full Time - Remote/BayArea in office - Compilers, Functional programming, Parsers, Parser Generators background. Ms, PhD preferred. Pay well. We’re pushing limits of Low Code, applied to Data. We work in Scala for Spark. Now that Snowflake, Databricks are established in cloud, we’re looking to win tooling layer. Email raj@prophecy.io
ibains··on Show HN: Prophecy: Low Code Data Engineering on Spark – free for small teams
At Prophecy, we're focused on making data engineers more productive. Visual drag-and-drop on Prophecy produces high quality code on Git.

We're releasing our SaaS product today - free forever for small teams (3 users). It is available on Databricks to start with.

Our existing product is used by a few Fortune 500 Enterprises and deploys within your network, and works with any Spark distribution.

Please try it! We're eagerly awaiting your feedback!

ibains··on From First Principles: Why Scala?
Prophecy.io - low code product for data engineering on Spark - is Scala based - we love it! We do compiler work - and packrat parser combinators are great.

Basic things are supported well. We use Play. We have GraphQL in web services - use Sangria. Slick FRM for database access is very clean. We use Akka for live web socket connections to Spark for interactive execution.

We extend Spark code from outside when needed - adding functions to classes in imported libraries.

It’s a learning curve for new engineers, but we hire the best and they like Scala after using it for a bit. SBT is confusing as hell, build times were painful but getting better. Debugging is sometimes hard (implicits)

One thing I’ve learnt us that if you’re handed a gun, you don’t have to start shooting at your feet just because you can. We used C++ at NVIDIA sensibly a decade back - using OOP but still avoiding horrible parts of C++. We use Scala sensibly now.

Can’t imagine another language that would work for us. I’ve used C, C++, F# before (building compilers and databases)- don’t know Kotlin. Only other language we use is Golang for our kubernetes operator (excluding UI)

ibains··on Advanced Compilers: Self-Guided Online Course
packrat parser combinators are recursive descent with infinite look ahead. That’s the problem, engineers are under educated
ibains··on Advanced Compilers: Self-Guided Online Course
There is no such variance and new fashions in compilers every year. There are hardware compilers and tools including language servers primarily. Move to cloud requires source to source compilers for data snd transforms.
ibains··on Advanced Compilers: Self-Guided Online Course
packrat parsers are fast (at expense of some memory)
ibains··on Advanced Compilers: Self-Guided Online Course
Yes, more validation - compilers as taught in universities need improvement.

So, a startup of 10 people should hire and train people for months. Raise a $2M seed round, have 12-18 months to show ProductMarket fit and as a founder teach all engineers their jobs? How many engineers go into compilers after school? So university education is all they know. The industry is built on qualified engineers. NVIDIA ceo was ahead of curve on graphics card, google founders ahead in search, why are compilers ones behind the curve?

ibains··on Advanced Compilers: Self-Guided Online Course
You might believe academia should not impart knowledge students require for their subsequent jobs. Ask the students why they go to university.

See what is taught in UCBerkeley with Spark and all coming out of there. I took a systems course with Eric Brewer - totally amazing - the context I got.

ibains··on Advanced Compilers: Self-Guided Online Course
We pay very very well, all my peers from NVIDIA are above 700k. Why would you presume anything about salary?
ibains··on Advanced Compilers: Self-Guided Online Course
I’ve worked on multiple compilers (optimizations expert) at MSFT on VS and CUDA and gave developed a DSL and worked on Database Compilers. I can’t hire compiler people with right skills. We’re building an IDE and those parsers are written differently, and we use Scala packrat parser combinators. These courses teach very little judgement or industry relevant stuff. When do you use packrat parser vs LALR vs LL? Good luck hiring an engineer with advanced compiler courses knowing any of this. I’d like to sit down all university professors who teach compiler courses and teach them a course on what’s relevant.
ibains··on To VCs: ELT is not the disruption – Data Engineering is
Folks, after Snowflake IPO and Dbt funding there is a lot of euphoria and FUD as I talked to some VCs. This doesn't match up when I talk to customers, so trying to clarify what's actually going on.
ibains··on A one-line change decreased our build times by 99%
Thus just shows how poor visibility into git is, I hope it gets better.

Building a product with poor visibility and ridiculing users for not knowing internals is the worst practice in Computer Science.

Hadoop did the same, and has set a record of fastest software to become legacy.

Super nice to see great comments here and the nice article.

ibains··on The next US president won’t be great for H-1B visas
You’re talking as if laws were sent by god, we made laws to serve us, not the other way around. We should change them to be sensible.

Also, if all businesses were required to be profitable from day 1, you’d have none. You spend on R&D to build a product, then sell it, then make money.

You know nothing jon snow

ibains··on The next US president won’t be great for H-1B visas
No, I’m saying that the parent comment suggesting bidding is not a good idea.

I’m hiring an H1B since that person has the right skills, and I’m hiring Americans as well (who might take lower salary and stock options initially). Putting something like auction for H1B means another competitive advantage to big tech over me.

>> Business model is broken

For startups, yes you’re working with constraints that seem impossible - you should encourage people to do hard things and build businesses, not give up - this is still US.

And yes I hired 10 engineers in India, and they’re paying taxes there. I’d rather hire them here and have them pay taxes here. In next few years, the founding engineers will start their own startups in India and not here, I want them to do startups here.

We’re screwing up by reducing supply of engineers, reducing supply of housing in US, and starving startup ecosystem.

ibains··on The next US president won’t be great for H-1B visas
I have a startup, 4 in US, 10 in India - top tier engineers. I just hired my first H1B - while we pay well - bidding and salary escrow will just favor the big 5 - they are starving startups of talent in bay area any way. Investors want me to get traction with $2M, our needed engineers are at $600-$800k comp (300 base) how do I make it work?

Regulation has unintended consequences. As an example trying to save retails investors from fraud means IPOs are fewer and large investors get all the growth.

Not to make this political, but highly educated and highly paid “employees” voting for “career politicians” and “bureaucrats” seriously overestimate their understanding of issues and competence. They are missing data points and should lean to minimum regulation.

Recent changes asking H1 to be paid >40th percentile of salary instead of 17th percentile seems sensible and enough.

ibains··on Nvidia to Acquire Arm for $40B
I love this, I was amongst early engineers on CUDA (compilers).

NVIDIA was so well run, but boxed into a smaller graphics card market - ATI and it were forced into low margins since they were made replaceable by OpenGL and DirectX standards. For the standard fans - they resulted a wealth transfer from NVIDIA to Apple etc. and reduced capital available for R&D.

NVIDIA was constantly attacked by a much bigger Intel (which changed interfaces to kill products and was made to pay by a court)

Through innovation, developing new technologies (CUDA) they increased market cap, and have used that to buy Arm/Mellanox.

I love the story of the underdog run by a founder, innovating it’s way to getting into new markets against harsh competition. Win for capitalism!

ibains··on Ask HN: Who is hiring? (August 2020)
Prophecy.io | San Francisco bay area | UI Architect | OnSite

Prophecy is a Data Engineering product on Spark/Airflow for Fortune 500. We’ve got a unique IDE based on visual studio code (browser) where you can do visual drag and drop, or code - and you can switch between the two instantaneously. Low code data engineering perhaps, along with our code editor that is specialized, and thus better for data engineering than vanilla VSCode.

Looking for a UI Architect to drive the UI in a team of 3. You have ideally worked on a complex product such as visual studio code, webflow or figma. You’ll drive the overall architecture of UI - and code - a lot.

Basics - React, Redux, GraphQL perhaps tests - Jasmine, Mocha,...

Send me a mail (raj@prophecy.io)

PS. We pay very well and hire only the very best.

ibains··on Ask HN: What are you using new programming languages for?
We’re using Go for writing a kubernetes operator to deploy our product.

Rest of our stack is in Scala - primarily since it is very good for writing compilers - we have a few in our product.

We’re a data engineering product - think Spark IDE.

ibains··on Spark vs. Snowflake: The Cloud Data Engineering (ETL) Debate
Very valid points: 1) Agree that Snowflake is far easier to use than Spark. 2) Agree that DBT is a great tool.

ETL workflows normally processing 10s of TBs and workflows with large and complex business logic is the context. With Spark code, you can break down your code into smaller pieces, see data flow across them, write unit tests, and have the entire thing still execute as a single SQL query.

Don't large SQL scripts become really gnarly for complex stuff - nothing short of magical incantations. I can't see data flow from a subquery for debugging without changing code.

Prophecy as a company is focused on making Spark significantly easier to use!

ibains··on Spark vs. Snowflake: The Cloud Data Engineering (ETL) Debate
Would love to get perspectives from HN community - did you decide between Snowflake and Spark for data engineering? Which one did you pick, and why?
ibains··on Show HN: Prophecy Spark IDE public beta
For the public beta and later our developer version, we only support Databricks.

However, for our Enterprise version, we install within your network and this supports EMR, Dataproc, Databricks.

This is only relevant where we are spinning up (and down) Spark clusters from within the IDE, you can always connect to an existing cluster from any distribution.

ibains··on Show HN: Prophecy Spark IDE public beta
Raj here, I'm the founder of Prophecy. Previously, I was the product manager for Apache Hive at Hortonworks (and wrote compilers for CUDA at NVIDIA, and Microsoft). The tooling for data engineers is really poor, and I saw many of my customers struggle with it. It's 2020, Elon just send people to Space station, and we can't build semi-decent data engineering tools?? - we've got to do better. So we'll fix it! Prophecy is unique:

- Unlike all existing ETL tools, we're code-first and Git centric. So we're perhaps we are the first Data Engineering tool.

- We want to give Apple like high quality design, and it-just-works experience.

We've built a unique CODE=VISUAL IDE where you can toggle between visual and code editors instantaneously. You can edit code, and the changes show up in visual graph, and vice-versa. So even if you use visual drag-and-drop view to edit a workflow, you're doing Git commits. You can just go to the underlying Git folder and do maven build. Same for unit tests. We've added one-click spin up of Databricks cluster from IDE and step-by-step execution.

We want to enable you to:

- Develop 10x better code for Spark

- Develop 10x faster on Spark

- Deploy to production 10x faster with CI/CD on Spark/Airflow.

For the beta, when you submit your e-mail, we create an account and send you login details. You can create Spark workflows, spin up clusters (in-built, on Databricks, free - please be kind with usage) and execute these. Please try out and give feedback, help us make data engineering better!

ibains··on [dead]
Super excited to share the public beta with HN community. We think Prophecy is a big step up in Data Engineering tools, and would love to get feedback on the product.

The public beta has Spark built-in, so you can just spin up clusters, please be kind with usage!

ibains··on Introduction to Apache Airflow
Has anyone used Enterprise schedulers - ControlM, Atom and know how it compares to Airflow?
ibains··on Maybe visual programming is the answer, maybe not
Maybe the path is to merge visual and code programming. We're doing that for data engineering - you can draw in visual editor, and press a button and you have code. So as long as you constrain the domain, it seems doable. https://prophecy.io/, https://medium.com/prophecy-io/spark-deserves-a-better-ide-9...
ibains··on Spark deserves a better IDE
Thanks for the great inputs, pulling Git info back into the IDE is a great idea. Will definitely add it!
ibains··on Spark deserves a better IDE
Excited about sharing the new interface we're building for Apache Spark.

We're also trying live screen gifs for the first time - quite an effort to get GIF size down.

ibains··on Ask HN: Are compiler engineers still in demand, and what do they work on?
I’ve worked in compilers (optimizations) at Microsoft (vc++) and NVIDIA(CUDA) - register allocators, schedulers and loop optimizers were super fun.

Then I worked on database (query) compilers. I managed Apache Hive at Hortonworks, and that has a compiler. These are very different beasts.

Now, my startup is working on code analysis and cross compilers for data engineering, including Spark.

Some of my colleagues are working in AI chip startups.

It’s hard to find high quality engineers in both kinds of compilers. Just like database internals engineers there is high demand.

Plug - I’m hiring at Prophecy.io for compiler engineers.

← PreviousPage 2 of 3Next →