HNHacker News
TopNewBestAskShowJobs

falaki

392 karma · joined June 20, 2014

submissionscomments
falaki··on Databricks response to Snowflake's accusation of lacking integrity
Apple is a big Deltalake (and Databricks) customer: https://www.youtube.com/watch?v=SFeBJxI4Q98
falaki··on Databricks response to Snowflake's accusation of lacking integrity
To be fair Apache Spark, which started long before either company existed, was built on the assumption that compute and storage should be separate. Unlike Hadoop, Spark did not come with any storage system and could read from any source.
falaki··on Databricks response to Snowflake's accusation of lacking integrity
That is exactly the role of tpc.org.
falaki··on Snowflake’s response to Databricks’ TPC-DS post
Linking to the discussion on the follow up from Databricks: https://news.ycombinator.com/item?id=29232346
falaki··on Databricks response to Snowflake's accusation of lacking integrity
tl;dr: The data warehouse company used a pre-baked TPC-DS dataset and claimed they have similar performance to Databricks. Turns out if you use the official TPC-DS data generation scripts, you get much worse performance.
falaki··on Apache Drill's Future
Spark community put a lot of effort into a feature-rich CSV data source. Often times the most challenging part of ingesting CSV is parsing it. There are many flavors each with different corner cases for NULLs, comments, headers, etc.
falaki··on Jsonnet – A data templating language
https://databricks.com/blog/2018/10/12/writing-a-faster-json...
falaki··on Uber S-1
Uber is clearly losing share to Lyft: "In 2017, our ridesharing category position in the United States and Canada was significantly impacted by adverse publicity events. Although the rate of decline in our ridesharing category position has since moderated, our ridesharing category position generally declined in 2018 in the substantial majority of the regions in which we operate, impacted in part by heavy subsidies and discounts by our competitors in various markets that we felt compelled to match or exceed in order to remain competitive."
falaki··on Getting started with Spark on AWS
Here is a much easier way to get started with Spark on AWS for free: https://community.cloud.databricks.com
falaki··on [dead]
A lecture by her in 2014. At that time she was suffering from cancer: <https://www.youtube.com/watch?v=tprlQMClSYQ>
falaki··on R vs. Python for Data Science [video]
Have you looked into SparkR?
falaki··on Iran's Great Wall Is Now Buried and Forgotten
Modern Iranians are to be blamed as well. Study of history is mostly overlooked. Archeological investigations are not funded. Part of it general mismanagement that applies to everything else. But the ideological regime in Iran is not keen about uncovering new historical facts that may alter its version of history.
falaki··on Setting a new world record in CloudSort with Apache Spark
Apache Spark is agnostic to storage layer.
falaki··on Samsung acquires Viv, a next-gen AI assistant built by creators of Apple’s Siri
A lot of the Siri IP belonged to Stanford SRI and Siri was basically licensing it. So when Adam left Siri and started Viv, Apple couldn't do anything. Also note that, Viv rarely talked about any details of their technology. Probably they were slightly afraid. At the same time Apple didn't want to pick a fight with a small startup and damage it's reputation. It might change now that Samsung is the owner.
falaki··on Samsung acquires Viv, a next-gen AI assistant built by creators of Apple’s Siri
I wonder if Samsung would ever put this on their Android Phones. I am confident it will not be able to compete with Google Now on the same device.

In addition to founders, Viv got a good number of early Siri engineers. From this and other hints I gather its internals are similar to first few versions of Siri. For example, they outsource the Speech component (something Apple still does for several languages). A consequence of that design is separating speech recognition from NLP, which makes it inherently more error-prone than Google Now.

falaki··on ParaText: CSV parsing at 2.5 GB per second
I wish there was more details on the benchmark. For example, I am interested to know if schema inference was turned on for spark-csv?
falaki··on Databricks Community Edition Is Now Generally Available
Direct link to the signup page, if you don't care about the press release: https://databricks.com/ce
falaki··on How we run Spark and Sqoop in production
Ever since using Apache Spark's Data Sources API was released, I have been relying no different Spark Data Source packages for my ETL jobs.
falaki··on U.S. Discloses Saudi Holdings of Treasuries for First Time
The question is, why does the U.S. need an ally in the Middle East? As oil becomes less and less strategically significant, and the Middle East region dries up [1], it will become less and less important. Do we desperately need an ally in Central Africa?

[1] http://www.climatecentral.org/news/middle-east-future-heat-h...

falaki··on Saudi Arabia and Iran: The Cold War of Islam
As an Iranian, I don't see it as clear-cut.

1. Climate change is accelerating and changing geopolitics of the region faster. Iran’s power and influence has been continuously decreasing in past 2500 years as the Iranian Plateau has got drier. As an example, Iran’s population centers have been moving north as time goes on. 2. Do not ignore role of management and human resources in power and development. In past 40 years Iranians have been making big management and strategic mistakes. Mistakes that engaged them in an eight year war with their neighbor — Iraq, put their interests at adds with world’s largest economic and military power — the United States, and now fighting a cold war with the majority Sunni in the middle east. What do all these have in common?

falaki··on LinkedIn CEO Jeff Weiner Gives $14M in Stock to Employees
Does anyone know how this $14M compares to what employees lost recently?
falaki··on Bokeh – a Python interactive visualization library
Fortran programmers find it much "easier" than other languages. You are arguing about a subjective matter here. The fact that there are so many R packages and users shows it is useful to a large enough subset of users in statistics/data science domain.
falaki··on Clash of the Titans: MapReduce vs. Spark for Large Scale Data Analytics [pdf]
Note that you can change Spark's aggregation algorithm to sort-based with a configuration.
← PreviousPage 3 of 3