1,241 karma · joined August 14, 2013
Spark is a data processing engine. Spark works with many persistent storage layers but provides none itself. So Spark can run on object storage (S3 etc), additionally it can run on HDFS (the Hadoop file system).
The right layer to compare Spark with is MapReduce or to compare its SQL engine vs other SQL engines that run on Hadoop (Impala, Hive LLAP, Presto).
YARN is used for resource scheduling in the Hadoop world but Spark has other options like its internal scheduler, Mesos, and now Kubernetes.
When people say Hadoop, the technical meaning is often HDFS + Hive + YARN + MapReduce (for legacy) + implicitly Spark on Hadoop. All Hadoop distributions support Spark.
So the comment Spark is faster that Hadoop is like saying Postgres is faster than Google Cloud.
Some of the graphics need to be improved. It would also be great to see why this is better than tradition BI tools (Tableau etc) and what your unique value proposition is.
It is good to see Sway is approaching 1.0 and that they are also producing a library https://github.com/swaywm/wlroots for building tiling window managers as I believe currently something like the 700 line monsterwm can't exist in the Wayland world due to the sheer amount of boilerplate and auxiliary code required.
It has recently added file list preview and tree view like functionality. Only missing on tags which are tricky because they break its pure markdown flat file philosophy.
Amazon used to have a great number of graduate preparation book lists which always included books such as Rudin's Principles of Mathematics Analysis, and Halmos' Finite Dimensional Vector Spaces. These classic maths books are brilliant but usually easier to understand if you already have some experience with the material.
Final advice is to find a study partner as it can be hard to track how you are going and keep motivated, especially without the instant feedback loop you get with programming.
I've done some work in Ceph and in my opinion it is too ambitious: block, object, and file system storage, custom raw partition format (Bluestore), along with a somewhat legacy C/C++ codebase makes the system fairly fragile. I would only recommend it if you "know what you are doing" or are willing to get some help from Red Hat.
The other comment I'd make is that 12TB unreplicated (assuming they are using 3x replication for Ceph) is actually not a huge amount of data and in my opinion a ZFS setup would be cheaper and more stable. It is not too challenging to do a HA ZFS setup, and also ZFS's many mirroring, scrubing, and checksuming abilities makes it very resilient. Copy on write and ZFS snapshots are also a great features that can save on disk usage.
I feel the sweet spot for Ceph is probably 50TB - 1PB where you need both block and object storage and are unwilling to use cloud solutions. Lower than 50TB and the overhead and risk of managing Ceph makes it less practical than traditional solutions. From the other direction CERN did testing up to 30PB with Ceph but they had to make significant code changes and had Ceph committers in their team. To compare Hadoop is running in clusters of up to 600PB (but you will be accessing the data very differently).
[1] https://cds.cern.ch/record/2015206/files/CephScaleTestMarch2...
Additionally NFS and SMB are very complex protocols and difficult to setup correctly. NFS without Kerberos provides only very basic whitelisting security. Adding Kerberos means you need active directory or something like FreeIPA and a scary amount of configuration (DNS server requirements, client side Kerberos config etc). I went down the whole Kerberos route and at the end decided mounting via ssh was far simpler and probably more secure. I would only trust a very technically competent vendor to do this well and don’t know if one exists.
I’ve seen this happen before in teams that code all day but don’t see themselves as developers. You need a instigate a cultural change at work where it becomes unacceptable to not apply modern development practices just because the team sees themselves as “quants” first.
1. Google Cloud Speech API
2. Microsoft Bing Speech API
3. IBM Watson Speech to Text
The ranking was as listed above but we had real challenges working with call-center audio recordings. The quality was less than idea but still very clear. We saw a huge reduction in accuracy compared to in-browser testing. Additionally Australian English is particularly not solved.
Because Google's API isn't currently doing speaker-detection, we looked at using Watson's speaker-detection as a secondary step but found it too complex and error prone. There is definitely room for a startup in this area and it also needs continued investment from the bigger cloud providers.
https://unix.stackexchange.com/questions/1489/is-mac-os-x-un...