nicklo

I use a plain pdf resume that is written in Latex. I host it on Overleaf so its always easy to update.

http://nicklocascio.com/resume.pdf

It absolutely is. DeepMind reported that 1 second of audio generation takes about 90 minutes to generate.

Super cool stuff in this paper.

At its heart, this is a new training architecture that allows parameter weights to be updated faster in a distributed setting.

The speed-up happens like so: instead of waiting for the full error gradient to propagate through the entire model, nodes can calculate the local gradient immediately and estimate the rest of it.

The full gradient does eventually get propagated, and it is used to fine-tune the estimator, which is a mini-neural net in itself.

Its amazing that this works, and the implications that full back-prop may not always be needed shakes up a lot of assumptions about training deep nets. This paper also continues the trend of this year of using neural nets as estimators/tools to improve the training of other neural nets. (I'm looking at you GANs).

Overall, excited to see where this goes as other researchers explore the possibilities when you throw the back-prop assumption out.

This comment is mind-numbingly anecdotal and runs wildly contrary to reality.

'"Happiness" at Mt Holyoke' - here's the list of majors at Mt Holyoke: https://www.mtholyoke.edu/acad/majors. 'Happiness' appears nowhere on the list.

"bunch of women who didn't even go to college" - females outnumber men in college 55/45.

Beyond the blatant disregard for facts, the comment is dangerously reductionist, attributing the diversity gap to a single cause- a lack of encouragement for females to pursue technical fields at a young age.

Yes, this is part of the problem, but no matter how much the talent pool increases, women are going to continue to leave tech at alarming rates until tech culture issues are improved as well.

Yeah their work is very cool! The two tasks are a bit different however. Their system is for the task of generating regular expressions given a set of positive and negative examples of what the regex should match. It uses genetic algorithms and other techniques to optimize and search for a regex that fits all the given examples.

In our case, we have no examples to test against, only a natural language (English) description of what the user wants the regex to do. This is an inference problem more than a search problem as we've got one shot to give our best guess without any tests to check against and modify our answer.

Thanks! Starting this work, we realized that large regex datasets (large enough to apply deep-learning to) were difficult to come by. So we came up with a methodology that allowed us to make a pretty decent-sized dataset for cheap. We are glad to share it :)

Sure!

Some positive ones:

1) Spot-on prediction:

  PROMPT: lines with 3 or more characters or lower-case letters

  PRED: ((.)|([a-z])){3,}

  GOLD: ((.)|([a-z])){3,}

2) Learned to generalize and produced a simpler regex:

  PROMPT: lines with a character and the string 'dog'

  PRED: .*(.)&(dog).*

  GOLD: .*((.)+)&(dog).*

3) Also learned to generalize and produced simpler regex without duplicate logic:

  PROMPT: lines not containing a letter

  PRED: .*~(([A-z])+).*

  GOLD: (.*)(.*~([A-z]).*)

4) Handling multiple references correctly:

  PROMPT: lines using 'su' after 'sun' or 'soon'.

  PRED: .*(sun|soon).*su.*

  GOLD: .*(sun|soon).*su.*

Though I find the mistakes interesting as well!

1) Issues counting properly:

  PROMPT: lines containing a 5 letter word beginning with 't'

  PRED: .*\bt[A-z]{5}\b.*

  GOLD: .*\bt[A-z]{4}\b.*

2) Misallocation of parenthesis (to be fair, the prompt is slightly ambiguous):

  PROMPT: lines with 'dog' follwed by 'truck' and a lower-case

  PRED: (dog).*((truck)&([a-z])).*

  GOLD: (dog.*truck.*)&(.*[a-z].*)

Hey HN! One of the authors here. This was a fun project to work on. Happy to answer any questions :)

Code + data here: https://github.com/nicholaslocascio/deep-regex

That account (bertilda-lovejoy) was made 10 minutes ago, so don't get too bent out of shape over it. Its likely a mens-rights red-pill-er intentionally misrepresenting feminist values by taking them to a tone-deaf extreme in a repo thats obviously not meant to be taken seriously.

Thanks for clearing that bit up. Glad that dietary considerations were made :)

Do you not find it unethical to serve your in-laws fake meat without informing them?

Wouldn't you be upset if your in-laws served you seitan that was actually chicken?

*edit: I bring this up because many people do not react well to meat substitutes. Gluten and soy intolerance affects many people and serving soy or gluten-based meat substitutes without prior informed consent can induce adverse reactions. I do believe that veganism in morally superior to carnivorism, but that still doesn't grant the right to trick others into consuming meat substitutes without their informed consent.

Hold up, an attempt to rectify systemic injustice is not reverse racism.

YC's doing a great thing here to support and foster underrepresented groups in the startup founder world. Being from an underrepresented group makes so many things harder because as a founder raising capital it, unfortunately, is often more about what you look like and who you know than your product or your vision.

The same way anyone gets good at anything - practice it. This may sound crazy but don't write it off. Go into bed during the day for 1 minute and don't fall asleep. Set an alarm and when it goes off, get out of bed. Repeat 5-10 times a day for a few weeks and getting out of bed when your alarm goes off will become habit.

Morning habits are important because when people wake up in the morning they are extremely tired which impairs their cognition and willpower, which often leads to bad decisions like the snooze button.

There are tons of ways to evaluate word vector quality! Word analogy tasks, word similarity tasks, contextual prediction tasks, etc.

This link contains a bunch of relevant evaluation datasets and benchmarks obtained using word2vec, GloVe, etc. You can evaluate your RNN-learned vectors and compare them to a traditionally trained word2vec-trained vectors. Link here: http://www.bigdatalab.ac.cn/benchmark/bm/Domain?domain=Word%...

For more background on evaluating word vectors check out these pretty great lecture notes from Socher's NLP class: http://cs224d.stanford.edu/lecture_notes/LectureNotes2.pdf

Also, here's the original papers from a few years ago that introduced many of these datasets and evaluation standards:

https://papers.nips.cc/paper/5021-distributed-representation...

http://www.cs.cmu.edu/~mfaruqui/papers/acl14-vecdemo.pdf

MetaMind papers are always pretty awesome. Couple of great highlights from this paper:

The visual saliency maps in Figure 6 are astounding and make their performance on this task even more impressive as they give a lot of insight into what the model is doing and it seems to be focusing on the things in the image that a regular person would use to decide on the answer. Most striking was on the question "is this in the wild?", and the saliency was on the artificial, human structures in the background that indicated it was in a zoo. This type of reasoning is surprising as it requires a bit of a reversal in logic to come up with this way of answering the question. Super impressive.

The proposed input fusion layer is pretty cool - allowing information from future sentences to be used to condition how to processes previous sentences. This type of information combining was previously not really explored, and it makes sense that it improves the performance on the bAbI-10k task so much as back-tracking is an important tool for human reading comprehension. Also its clever that they encode each sentence as a thought vector before compositing so they can be processed both forwards and backwards with shared parameters- doing so on just one-hot words or even ordered word embeddings would require two vastly different parameters since grammar is wildly different when a sentence is reversed.

Lastly, on a side note, if 2014 was the year of CNN's, 2015 the year of RNN's, it looks like 2016 is the year of Neural Attention Mechanisms. Excited to see what new layers are explored in 2016 that will dominate 2017.

Wonderful write-up. Posts like these give just the right amount of detail and high-level insight.

I especially appreciated the linear flow of the approaches section. The successes and roadblocks presented provide useful context into why certain decisions were made, which is too often missing from many academic papers.

From a technical standpoint, the paper is somewhat lacking. It's a standard Markov-chain generator with an additional prior defined on the cohesiveness of the generated text.

I'm guessing all the press is due to it being US election season, but this is really no different from King James Programming (http://kingjamesprogramming.tumblr.com), which was entertaining, but nothing revolutionary.

Cool! Let me know if you run into any issues.

Thank you for this. I see now how my wording was unclear, but this is exactly what I had meant. Sorry for the confusing word choice.

This strategy is also encouraged by best iOS development practices (main thread never works on GUI).

Edit: 'main' as in the one doing the one doing the work, in the same spirit of the original post. In truth, the true main thread that runs in main() is where all the iOS GUI work happens. Apologies for the confusing wording.

An Embedding layer by itself will try to learn word vectors from the data you train it on (error gradient will propagate back to embedding layer, update the vector weights). Word2vec and word vectors are only really useful with tons of training data to learn good embeddings. I believe OP is referring to using Google's pre-trained word-vectors (trained on massive amounts of text [100 Billion words]). Can be found here: https://code.google.com/p/word2vec/

This is pretty straightforward to implement in keras, you just need to supply pre-trained word-vectors weights to your embedding layer.

  Embedding(vocab_size, 300, weights = [word2vec_weights])

Where 'word2vec_weights' is a numpy matrix with shape (vocab_size, 300).

This paper focuses on statistical parametric speech synthesis (SPSS). SPSS is only 1/2 of the text to speech problem.

SPSS is the problem of going from linguistic features, phenomes, etc, to speech audio. These features are more or less golden, either derived from the audio itself or hand-labeled. So things like tonality, cadence, emphasis on words is already encoded as features which is why these samples sound so good.

Deriving these features from pure text is very hard, and this failing is the main reason most text to speech systems sound so dull and tone-dead.

That being said, these results are seriously impressive, sounding very natural. Would love to see someone try and train an end-to-end system from pure text to speech. I think we'd see some big improvements like what Baidu has done for end-to-end speech to text.

Went through a similar process when I cancelled by Creative Cloud. I got an email that my credit card had expired, so I put in the details for the new credit card. I hadn't even realized I was still paying for Adobe CC, so I then called to cancel the service. Support agent went through this same script, I was forced to pay the absurd termination fee and thought that was the end of it.

I got an email the next day thanking me for updating my Credit Card information and that my Adobe Creative Cloud subscription had been renewed because the charge on the new credit card info went through. I had to call Adobe yet again and jump through through many, many hoops but eventually got it taken care of.

What a terrible experience, I'm going Sketch all the way from here on out.

I'm a bit irked by the article's original title: "What's he building in there?"

The phrase trivializes the work of a large team of various genders and backgrounds working together on a massive research initiative to extend the human lifespan and makes it seem like a whacky project by one hacker dude in his garage. Come on.

Biology is one of the most team-oriented, least male-dominated, best well-balanced STEM fields and science journalists are still writing titles like "What's he building in there?"

Any reason for the focus on old symbolic-AI techniques? Things like expert systems have fallen out in favor of learned machine learning models. Yes, machine learning is 'popular' and 'trendy', but it also happens to work better and require less engineering effort to build and maintain than symbolic-AI systems.

Is there a market for traditional AI techniques as a service? Or maybe the wording is just throwing me off? The word 'agents' in particular is used a lot in 60's AI papers (Minsky's influence), so maybe I'm misinterpreting?

The space for AI-powered venture-backed companies is really booming.

Full list of companies mentioned in the article:

Enigma - http://enigma.io/

Sift Science - https://siftscience.com/

Ravelin - https://ravelin.com/

H2O.ai - http://h2o.ai/

Seldon - http://www.seldon.io/

SigOpt - https://sigopt.com/

Gluru - https://gluru.co/

x.ai - https://x.ai/

SwiftKey - https://swiftkey.com/en

Matternet - https://mttr.net/

SkyCatch - https://www.skycatch.com/

DeepMind - http://www.deepmind.com/

Vicarious - http://www.vicarious.com/

Scaled Inference - https://scaledinference.com/

Sentient Technologies - http://www.sentient.ai/

MetaMind - https://www.metamind.io/

Unbabel - https://unbabel.com/

Ravelin - https://ravelin.com/

Signal - http://signalmedia.co/

Sano - http://www.sano.co/

Deep Genomics - http://www.deepgenomics.com/

Atomwise - http://www.atomwise.com/

Flatiron Health - http://www.flatiron.com/

Enlitic - http://www.enlitic.com/

Zebra Medical - https://www.zebra-med.com/

DueDil - https://www.duedil.com/

Premise - http://www.premise.com/

The applications for this in the advertising industry are exciting and a tad bit scary.

I can imagine a feedback system that takes the output of this algorithm and modifies the image slightly according to the error gradient to optimize the MEM-score of the brand/item being advertised in the photo. Then it could feed the new image back in and repeat like the Deep-Dreaming algorithm.

side-note: I'm so happy that CSAIL is finally embracing deep learning. I'm an undergrad at mit and this semester was the first that deep learning was a major part of both the computer vision class and the nlp class.

OpenAI seems to be taking a different approach from OpenCog. OpenCog aimed build a monolithic framework for many existing AI and machine learning techniques. This has been done many times before.

OpenAI is more about exploring new research areas and pushing the cutting edge, while publishing papers and sharing code along the while. Both are admirable goals, but what OpenAI is aiming for has never been attempted before.

Very excited to see what comes of it!

Very exciting announcement. I never quite understood all the criticism and doubt of DWave. Do people think Google wouldn't do due diligence before investing hundred of millions of dollars in the tech? Anyway, happy to see this announcement and looking forward to hearing more in the future.

https://www.google.com/#q=SEC+Steps+Up+Probe+of+Pre-IPO+Shar... to bypass paywall.