325 karma · joined August 22, 2012
Ex-NASA scientist. Triathlete. Gamer.
1. Open a Google Chrome Incognito or Sarafi private browsing session
2. Paste the URL
3. #boom
Anyone working on large scale data problems or doing advanced analytics (Spark, R, Python, etc.) would appreciate having more memory to allocate to their problems.
The analysis in this blog underestimates the complexity of various compute environments.
1. How should we receive costly research that took special equipment and lots of time to develop and cultivate? I.e., CERN?
2. A lot of research is published, ignored, and then rediscovered. In this case, we may want to accept the research until it cannot be repeated (i.e., in another journal publication).
3. Reviewers of academic publications probably are not qualified or have the time to recreate all scientific research.
4. Isn't the academic system at its core kinda... broken?
I hate when articles make these ridiculous claims in order to inflate their credibility. Are there problems the security community needs to address before using technology to vote in a democracy? Yes. Is online voting a danger to democracy? No, but there are bigger problems that need to be solved before online voting should be implemented.
Point: I just tried to temporarily remove a freeze on my credit report after the Office of Personal Management (aka, the office for federal employees) lost all of my PII in a large-scale hack that occurred more than a year ago. For those of you interested, all of my credit is essentially frozen indefinitely as a result of this hack. Now, in order for me to validate my own identity, I had to reproduce (sometimes unsuccessfully) a series of data points that anyone with a hard copy of my credit report combined with my OPM breached data could reproduce. The real issue is the fallacy that a human being is uniquely identified by a set of data points (paper or otherwise). This is fundamentally the issue that must be overcome before we can breach issues like online voting reliably. We continue to create systems based on this fallacy of personal identification, and it is creating more problems than it is solving. Again, paper or otherwise.
Good luck!
Congratulations Pokemon! Thanks for continuing to bring out the kid in me!
val myDF = sqlContext.read.format("com.databricks.spark.csv") //allows you to read a csv file (for simplicity)
.option("header", "true")
.option("delimiter", "\\t")
.option("mode", "PERMISSIVE")
.option("inferSchema", "true")
.load(filename)
.where(filter_query)
.cache()
myDF.registerTempTable("myDF")
sqlContext.sql("SELECT COUNT(*) FROM myDF").show()
sqlContext.sql("SELECT COUNT(*) FROM myDF where filter_query").show()Native refers to (I think) the following issues: https://issues.apache.org/jira/browse/SPARK-12785 https://issues.apache.org/jira/browse/SPARK-8641
Code generation is enabled by SPARK-8641, but not sure exactly what it entails. I think it is related to some of the RDD transformation/action merging they do to optimize runtime operations in 2.0.
Your thoughts?
But you can always download and build the source!
I've done basic game development before (built centipede). Really like the design considerations involved in VR. Would love to start playing with that.
I wish I had a collaborator or someone I should bounce ideas off of.
My background is in advanced mathematics and I've gravitated towards data science and engineering. Would love to be a part of the game development process, even if not developing directly.
If I were to use my skill set in game development it would probably be in MMO games, or something with lots of active users interacting.
I have been following the handmade quake blog. I like it, just don't have enough time to really do anything meaningful.
Advice?