HNHacker News
TopNewBestAskShowJobs

vtuulos

780 karma · joined February 28, 2011

Helping companies build real-world ML/AI systems.

CEO/co-founder/customer support representative at https://outerbounds.com

I started https://metaflow.org at Netflix where I led ML infra for 3+ years. I wrote a book about my learnings:

https://www.manning.com/books/effective-data-science-infrastructure

Earlier projects:

http://traildb.io

http://tuulos.github.io/sf-python-meetup-sep-2013/

http://discoproject.org

submissionscomments
vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
Metaflow comes with a built in scheduler. If your company has an existing scheduler, e.g. for ETL, you can translate Metaflow DAGs to the production scheduler automatically. This is what we do at Netflix.

We could provide similar support e.g. for Airflow, if there's interest.

For monitoring, we have relied on notebooks this far. Their flexibility and customizability is unbeatable. We might build some lightweight discovery / debugging UI later, if there's demand. We are a bit on the fence about it internally.

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
note that Metaflow comes with a built-in DAG scheduler, so it is perfectly usable without a centralized scheduler like Meson.

Integrations to Airflow or StepFunctions are on the roadmap, depending on what seems to resonate with people outside Netflix.

vtuulos··on Metaflow, Netflix's Python framework for data science, is now open source
hey, I'm one of the authors of Metaflow. Happy to answer any questions! Netflix has been using Metaflow internally for about two years, so we have many war stories :)
vtuulos··on Ursa Labs: an innovation lab for open source data science
This is very exciting, thanks Wes! A robust, high-performance, polyglot, in-memory runtime for data(frames) would be extremely useful.

The remarks about challenges related to OSS maintenance/innovation were spot on too.

vtuulos··on Streams: a new general purpose data structure in Redis
Redis Streams could be a nice real-time counterpart to http://traildb.io: For instance, use Streams in Redis to record data in real-time and periodically store it in TrailDBs for long-term archival and analysis.
vtuulos··on Show HN: Infreqdb – S3 backed key/value database for infrequent read access
In our case all blobs in S3 are immutable, so no need to invalidate anything.
vtuulos··on Show HN: Infreqdb – S3 backed key/value database for infrequent read access
We use a similar approach with TrailDB. You can make S3 access totally seamless with user-space page fault handling, which is pretty cool: http://tech.adroll.com/blog/data/2016/11/29/traildb-mmap-s3....
vtuulos··on Ask HN: Who is hiring? (March 2017)
AdRoll | San Francisco | On-site/remote | Full-time

If you like developing open-source code, languages such as Python, JS, C, D, Lua, Erlang, AWS, petabytes of data, and distributed low-latency systems, this may be your dream job.

This time we are particularly interested in finding data scientists and experienced Erlang developers / tech leads. This is a really unique opportunity to get to work with a massive scale (thousands of instances on AWS), low latency (real-time bidding, real-time machine learning), mission-critical systems (this is how we make money).

Learn more about us here

http://tech.adroll.com/blog/

I am happy to tell you more over coffee in SF or by email, ville@adroll.com

vtuulos··on Show HN: PumpkinDB, an event sourcing database engine
I am a big fan of this approach: Low-level, high-performance, immutable-by-default database engine as a building block, with bindings to multiple languages for easy application development.

TrailDB (http://traildb.io), which has many elements of event sourcing in it, follows this philosophy and it has proven to be pretty successful for its intended use cases.

I was delighted to notice that PumpkinDB has an imperative query language inspired by Forth. We recently open-sourced a similarly imperative query language inspired by AWK, http://github.com/traildb/reel :)

I will definitely follow PumpkinDB with great interest!

vtuulos··on Beringei: A high-performance time series storage engine
Great to hear that!

We would love to hear how you are using TrailDB and what could be improved for your use case. Feel free to open issues in GitHub or drop by at our Gitter channel, https://gitter.im/traildb/traildb

vtuulos··on AWS Batch – Fully Managed Batch Processing
First time I hear about Elasticluster. Regarding cluster provisioning, auto-scaling is critical for cost and convenience. We managed to do this with a simple job queue that outputs a CloudWatch metric that in turn controls an auto-scaling group of spot instances. Works like a charm but AWS Batch should make this redundant.

We took a look at Airflow, which looks nice, but we haven't had a compelling reason not to use Luigi.

vtuulos··on AWS Batch – Fully Managed Batch Processing
Thanks! Pachyderm's version control over datasets is a killer feature.
vtuulos··on AWS Batch – Fully Managed Batch Processing
We have been using this architecture for containerized batch processing for more than a year:

http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-d...

It is very convenient to be able to define any jobs as Docker containers, using the best language for the task, and define dependencies between their inputs and output explicitly, not being tied to any particular paradigm like MapReduce.

I am more than happy to let AWS handle plumbing for this instead of having to maintain and operate our custom solution.

vtuulos··on Beyond Nokia: A love story
In 2006-2007, just before iPhone was released, we tried to pitch Nokia to adopt Python as the main language for mobile application development. We even wrote a book about it:

https://www.amazon.com/Mobile-Python-prototyping-application...

Python for Symbian (S60) was amazing - arguably the best way to develop smartphone apps at the time. It made you orders of magnitude more productive than native, convoluted version of C++. It came with bindings to camera, accelerometer etc. The underlying vision was to enable large masses of people to develop mobile apps quickly and thus unlock Nokia's capable hardware that was buried under a clunky OS.

A Python-based user-driven app ecosystem might not have killed iPhone but it might have filled some of the vacuum that gave rise to Android. Nokia was way ahead of its time with many ideas.

vtuulos··on US launches next-gen weather satellite that will revolutionize forecasting
Fun question, even if it wasn't a serious one. Since I happen to work at a company whose business is related to ad placement prediction, I can try to answer.

In our case, we have 1000-2000 instances (the number is elastic, because AWS) handling real-time ad auctions and related data processing. Since we are just one of the many players in the industry, and far smaller than the largest ones (GOOG, FB), I am pretty sure in aggregate more computing cycles are used to optimize ads than oil or weather.

Crazy.

vtuulos··on Ask HN: Who is hiring? (October 2016)
AdRoll - San Francisco

If you like developing open-source code, languages such as Python, JS, C, D, Lua, Erlang, AWS, petabytes of data, this may be your dream job.

AdRoll is one of the most widely used adtech companies, already producing $100M+ in revenue with a small engineering team.

Recently we open-sourced a core part of our data infrastructure, TrailDB:

https://news.ycombinator.com/item?id=11762801

http://tech.adroll.com/blog/data/2016/05/24/traildb-open-sou...

Machine learning is also close to our hearts:

http://tech.adroll.com/blog/data-science/2015/08/25/factoriz...

As well as modern UIs built with React.JS

http://tech.adroll.com/blog/frontend/2015/11/05/rollup-share...

I'm happy to tell you more by email (ville@adroll.com) or over coffee in SF.

vtuulos··on Ask HN: Who is hiring? (June 2016)
AdRoll - San Francisco

If you like languages such as Python, JS, C, D, Lua, Erlang, AWS, and/or petabytes of data, this is your dream job.

AdRoll is one of the most widely used adtech companies, already producing $100M+ in revenue with a small engineering team.

Just last week we open-sourced a core part of our data infrastructure, TrailDB:

https://news.ycombinator.com/item?id=11762801

http://tech.adroll.com/blog/data/2016/05/24/traildb-open-sou...

Machine learning is also close to our hearts:

http://tech.adroll.com/blog/data-science/2015/08/25/factoriz...

As well as modern UIs built with React.JS

http://tech.adroll.com/blog/frontend/2015/11/19/rollup-major...

I'm happy to tell you more by email (ville@adroll.com) or over coffee in SF.

vtuulos··on TrailDB – An Efficient Library for Storing and Processing Event Data
TrailDB is designed for discrete events (think JSON objects) rather than continuous metrics (like CPU utilization).

Most time-series databases manage to store a large number of data points by aggregating data over time. Discrete events can't be aggregated. Instead, TrailDB leverages predictability of events to compress data over time.

vtuulos··on TrailDB – An Efficient Library for Storing and Processing Event Data
Good question. Judy arrays tend to be really efficient for low-level mappings like the 128-bit -> 64-bit map (https://github.com/traildb/traildb/blob/master/src/judy_128_...) that we use to keep track of UUIDs.

It would be interesting to benchmark Judy against another similar well-optimized data structure.

vtuulos··on TrailDB – An Efficient Library for Storing and Processing Event Data
You can find the R bindings now at https://github.com/traildb/traildb-r
vtuulos··on TrailDB – An Efficient Library for Storing and Processing Event Data
We were just talking about it earlier today. It should be easy to do with https://github.com/node-ffi/node-ffi

Contributions are welcome :)

vtuulos··on TrailDB – An Efficient Library for Storing and Processing Event Data
TrailDB is optimized for a very specific data model (http://traildb.io/docs/technical_overview/#data-model) which allows it to do compression that you couldn't do with other data layouts.

TrailDB is a C library, Parquet and Avro are not. Depending on your use case this might be a pro or a con.

vtuulos··on TrailDB – An Efficient Library for Storing and Processing Event Data
It is coming soon, sorry for the delay!
vtuulos··on TrailDB – An Efficient Library for Storing and Processing Event Data
Thanks for you kind words!
vtuulos··on TrailDB – An Efficient Library for Storing and Processing Event Data
Pipelinedb.com is a great product and very complementary to TrailDB.

PipelineDB is all about querying streaming, real-time data in SQL whereas TrailDB is great for computationally intensive analysis of historical data using any programming language.

vtuulos··on TrailDB – An Efficient Library for Storing and Processing Event Data
Our infrastructure is in AWS, so S3 was a natural choice for us. You can easily use any other object store or filesystem with TrailDB.

We use Kinesis amongst other things to stream raw data to S3.

vtuulos··on TrailDB – An Efficient Library for Storing and Processing Event Data
Thanks! I hope we can open-source the rest of the stack now that the core libraries are out.

Btw, I am happy to answer any questions about the project here.

vtuulos··on Ask HN: Who is hiring? (May 2016)
AdRoll - San Francisco

If you like languages such as Python, JS, C, D, Lua, Erlang, AWS, and/or petabytes of data, this is your dream job.

AdRoll is one of the fastest growing adtech companies, already producing $100M+ in revenue with a small engineering team.

To give an idea how we work, recently we launched a new product on top of a novel data processing pipeline on AWS using Docker, read more about it here:

http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-d...

and more about our approach to data science, see here

http://tech.adroll.com/blog/data-science/2015/08/25/factoriz...

http://tech.adroll.com/blog/rtb/2016/04/29/collider.html

I'm happy to tell you more by email (ville@adroll.com) or over coffee in SF.

vtuulos··on Ask HN: Who is hiring? (March 2016)
AdRoll - San Francisco

If you like languages such as Python, JS, C, D, Lua, Erlang, AWS, and/or petabytes of data, this is your dream job.

AdRoll is one of the fastest growing adtech companies, already producing $100M+ in revenue with a small engineering team.

To give an idea how we work, recently we launched a new product on top of a novel data processing pipeline on AWS using Docker, read more about it here:

http://tech.adroll.com/blog/data/2015/09/22/data-pipelines-d...

and more about our approach to data science, see here

http://tech.adroll.com/blog/data-science/2015/08/25/factoriz...

http://tuulos.github.io/pydata-2014/

I'm happy to tell you more by email (ville@adroll.com) or over coffee in SF.

vtuulos··on Luigi vs. Airflow vs. Pinball
No, we haven't used Luigi with Hadoop. For batch processing we use containerized jobs with a simple job queue, like described in the blog article.
← PreviousPage 3 of 6Next →