ibains

Prophecy.io - Cloud Native Data Engineering.

Hiring for Spark (database) internals and Scala.

Offices in San Francisco, Gurgaon, Bangalore. Onsite.

Contact Raj at +1.415.310.1638

Hi everyone! We write parsers and didn't find a good post showing how to solve practical problems using parser combinators in Scala. Sharing our experience and hoping to learn from yours.

I worked on compilers at Microsoft, NVIDIA and there are few jobs for traditional hardware compilers. Some new hardware is being designed though, most fun was new optimizations specific to GPUs different caches such as loop optimizer and rematerialization in register allocator.

Then I’ve worked on query optimizers for databases and they’ve got completely different technology and papers. Here one can extend Apache Spark, they give lots of interesting extension points.

Also, these days my startup is building parsers in Scala for DSLs and if performance is not critical, I’m loving Packrat parsing in Scala (parser combinators), this is way easier and fun. Interesting tooling can be built in Scala, you can also use Scala macros, get access to Scala compiler AST. This kind of work around data might have applications for more engineers.

We're a scala shop, we get a functional language with features such as parser combinators (packrat parsers) - super useful for building DSLs. Functional-Relational-Mapping (for comprehensions) for SQL. We're able to get good web services with play, with easy to write asynchronous code. Plus all of it runs on robust JVM. We love it, we'd not change it for us, hype or no hype.

Hi everyone! in trying to find product market fit, I've learned the great value of customer discovery. I've written it how I'd share with a friend starting her own startup. Look forward to learning from other people's experience as well!

This is the workflows written in legacy formats - Ab Initio, Informatica, IBM DataStage - this is a $10B+ industry - so lots of usage out there. They typically have graph formats and Domain Specific Languages to specify transformations. We reverse engineer them. Enterprises have 10s of thousands of these workflows and getting out of legacy lock-in is hard!

The variation of complexity of transforms across Enterprises and teams within Enterprises is quite large.

Spark and Hive (that I was product manager for, and used by hundreds of Enterprises) are used quite heavily in data engineering. Many transforms fit well into relational/set based model.

In your particular case, perhaps due to the nature of your transforms, it might not have been the best fit.

Also, if your data is small, it's not always a great fit. However, in many large companies, if they have 500 transforms and 30 are small - there is simplicity in using a single technology.

Well, simplest way to put it is - you get an Data Engineering (ETL) product - that'll move your 10s of thousands of ETL workflows from legacy ETL product to Spark, give you a full stack ETL product on Spark, and move you from on-premise to cloud.

Plus it has improvements along multiple dimensions such as interfaces - that developers care a lot about!

We're not adding a layer on top of all big data platforms, instead we work on Spark directly. This has advantages, disadvantages.

When you use our product, your code looks very similar to writing Spark code in IntelliJ. You get visual workflows along with it.

Our users are not having to learn or use another layer of abstractions that locks them into our API, and we avoid performance

Apart from interface, we automatically "convert" your legacy code into Spark & "migrate" these Spark workflows from one cluster to another. You can move workflows from private cloud to public cloud, or to another datacenter.

Foremost, we're focused on Transforms - helping users create and manage complex ETL transforms - this is your business logic.

One is the category of schedulers - airflow, astronomer - these are focused on scheduling these Transforms, so these are very different (we integrate well with airflow).

Then there is the NoCode category of transform services on cloud - these focus on simple movement of data - datacoral, segment, fivetran and all might fit here.

We're quite different from both!

We're focused on complex transforms for high volume & performance workloads - something you'd find in Enterprise core production ETL workflows.

We're a complete replacement for ETL products - you'd use us to replace an Ab Initio, Informatica, IBM Datastage - we'd move your workflows through automated conversion.

You can program using visual drag-and-drop or write pure Spark Scala/Java/Python code, they are equivalent in our system. You get configuration management, lineage, metadata management. We're not tied to any private/public cloud - so you can use us with any Apache Spark - every public/private cloud has it.

This is quite different from these products! Different focus and level of abstraction.

I'd love to understand what your preferred interface looks like, and get your input in building the interface, please send me a mail at raj.bains@prophecy.io

Hi Everyone! Super excited for our first release!!

I was product manager for Apache Hive at Hortonworks and earlier an early engineer at CUDA at NVIDIA - focusing on compiler optimizations.

As product manager, I saw so many customers struggle with Hive/Hadoop. So, I decided to build a product first company - that just works to solve real (and sometimes unsexy) pain points of customers.

We want to support entire Enterprise journey to Open Source (Apache Spark) and then to cloud (Kubernetes).

What I'm personally excited about is that with some compiler magic - I can make code and visual interfaces both work together - making all developers happy!

I'd love to hear what you think, what you wish we'd build, and I'll be here to answer any questions!

Architect, Principal Engineer | Seed stage startup | San Francisco, Bangalore, Gurgaon. ONSITE -

We’re building Cloud Native Dataflows on Spark and Kubernetes to replace ETL.

Need expertise in one or more of Programming Languages/Compilers, Distributed Systems, Distributed Database Internals.

Stack is React, Scala

Contact me (Founder) raj dot bains at simpledatalabs dot com