Hacker News Reader

Comment by aerovistae | Hacker News Reader

As a non-ML practitioner who has an interest in the field, I have a question I was hoping someone could answer.

Looking at the page this links to, it gives pretrained models for "common use cases", among other things. Can you actually do anything interesting or novel with those? I feel like any meaningful new use of ML will not be possible with such "beginner building blocks" and you would just wind up with a weaker version of existing offerings. Could someone give me any examples of meaningful ways this can be used?

View on HN

You are correct that all commoditized AI currently cannot do anything completely novel.

They ship a network to convert landscape photos to landscape paintings, for example. You can retrain that with limited effort and knowledge by using their Collab notebook. That way, you can make an AI to convert photos of cats to photos of dogs. But you wouldn't be able to create an AI to convert between cats and rabbits, for example, because those are too different for that AI architecture to tolerate.

Similarly, they ship a classifier to detect objects of some categories based on surface texture. You can retrain that to detect different categories of things. But if someone paints a chair to look like elephant skin, the AI will be confused and short of inventing a new architecture yourself, there's nothing you can do to fix it.

Plus in any case, you will need to have large high quality datasets for your retraining. The biggest cost in building a bird classification app isn't the one ML employee that creates the final AI, it's hiring biologists to correctly tag thousands of photos to train the AI on.

That's why many AI research nowadays tries to offload the data generation to random people on the internet. For example, recaptcha, or crawling Flickr for that faces dataset.

Edit: since it was brought up in the comments, the retraining that I mentioned can be done only on parts of the model, which is then called transfer learning and can save you some computing time.

But, you'll need to determine if the first feature layers of the original model are suitable for your new classification task. Even experts sometimes guess wrongly here, so as a beginner, I'd say you can only pray and hope.

"But you wouldn't be able to create an AI to convert between cats and rabbits"

I reckon you probably actually could with some of the more recent results.

I am not aware of any model on the TensorFlow Hub which could do that.

I also believe that I haven't read a paper with a suitable architecture yet.

Which one are you referring to?

CycleGAN: https://github.com/junyanz/CycleGAN

CycleGAN is exactly the one from the Google colab notebook that failed for cats to dogs. It'll produce cats with a dog texture, but with cat anatomy.

TensorFlow Hub is probably outdated, as usual. They're still advertising ResNet50 for goodness sake.

Try this: https://github.com/NVLabs/FUNIT (2019)

Or this: https://github.com/clovaai/stargan-v2 (2020)

Either of those should probably work for cats to rabbits.

I disagree.

Both of them massively reduce the complexity by working on position-aligned images of purely the head. I would estimate that even a picture of a cat from the side will already be too much for this network to handle.

These kind of networks learn their feature space mapping by treating the input data set as continuous. So for them to learn a good mapping from cat to dog, it would need to also see photos of an animal that is half cat half dog. If I had to train this case for work, I guess I'd try to go through baby pictures. Dog -> baby dog -> baby cat -> cat. That might work if baby cats and baby dogs look similar enough.

I don't think pretrained models are "beginner building blocks". They provide a kind of "base model" that can be fine tuned to your specific need. The advantage of not training the full model means you can save many resources (in design time and computation).

There are many boring but meaningful tasks in which this can be used. For example, I'm sure many industries could be benefit from image classification for very specific cases (e.g., fruit categorization). In those cases, you are not interested in the classification of "general" objects such as car, person, bike or horse (as provided by MobiletNet pretrained model), but you can use that model as base to classify different categories of fruit.

Therefore, you are right that they might not be very useful to build new ML algorithms or network architectures. But they are useful to build specific (and novel) uses of current neural networks.

Re-training! The generalised models are... Generalised. They have learned to pick up details from a vast, varied data set.

You can reuse that, and fine tune to the model to your specific data set, which could save you days worth of computation, and also decreases the data set size requirement on your end.

Sure. The innovation often comes from how you use the model, not trying to build new ones (although that happens too).

They have pose detection: https://github.com/tensorflow/tfjs-models/tree/master/posene.... One day during a talk with someone who works in aged care I worked up a demo of fall detection using this.

That sounds pretty interesting. Do you have notes you can share?

And some years back I use posenet in node.js, just for its head tracking, while exploring 3D UIs. Because it was fairly robust, at least as long as an eye was visible. Even now, it degrades more gracefully with heads turned away than some face nets. Though its positions are a bit noisy, which hurt multi-camera-parallax 3D pose estimation.

There is a difference between consuming models and training them yourself -- I could be wrong but with tensorflow.js I think it is much more geared toward consuming models in the browser, rather than training novel ones yourself. Seems like you can customize an existing model via transfer learning, which may or may not be good enough for your usecase, but even then I feel like basically the training side of it is second rate. But you could potentially make _predictions_ with a really sophisticated/cutting edge model in JS -- its not the prediction part that takes enormous compute, it is the training part

You can also build and train models in js. This is one of the features I'm looking at. Currently there is a lot of information websites are throwing away or they're using centralized aggregation mechanisms to provide personalization but Tensorflow.js changes that dynamic.

I think small AI models that run completely in the browser and provide personalization by learning from how the user interacts with a given website are the future. This empowers the user and puts them in charge of how their data is used. The example I mentioned previously to demonstrate this was about ranking new submissions on HN: https://news.ycombinator.com/item?id=23407549. I'll quote the relevant part

> TensorFlow.js is a pretty nifty piece of software and it's underutilized. If the model parameters can be stored in IndexedDB then users could train TensorFlow.js based site augmentation to suit their own needs. For example, what if HN had a TensorFlow.js model for ranking new submissions based on the user's preferences? This model could be trained like a spam filter and would eventually learn the types of articles that someone likes to see but they would be in charge of the model's evolution and so would be empowered to use it however they saw fit. Maybe I don't care about politics then my model parameters would eventually converge on downgrading all political posts and the more technical submissions would rise to the top based on how I upvoted new and front page submissions.

Cool idea, but why not have that client side model operate across websites -- if I switch from hacker news to reddit why not carry over that data to figure out what I will like on the separate platform?

Excellent idea. Yes, you could do that as well. Nothing would prevent copying the parameters and models for one site and using at another one since at the end of the day it would all just be data controlled by the user.

One can even imagine a decentralized sharing mechanism where users can create ensembles of models by combining models trained by different users.

All the building blocks are there. Just requires mindshare and a few killer applications.

This is a cool idea, but requires some hooks. Otherwise, you also need api access and remake the UI. I guess you could intercept API calls with a browser extension, but having a hook for you personalization so would be great!

Here is an example thing I made using one of the pretrained mdoels (MediaPipe Facemesh):

https://github.com/paruby/snake-face/

I'm sure that many people would not class this as interesting or novel, but it demonstrates that at least some of the building blocks can actually be used to make actual things.

There is a process called transfer learning where you take a model that has been trained on a large data set and then modify the last few layers of the neural network by training them on your specific/smaller data set. Not sure if there are any good resources for beginners but I learned about the technique from fast.ai videos.

It depends on your definition of "novel." I've seen some of these base models for tfjs used in several neat hackathon projects ranging from music generating web apps to "hot dog, not a hot dog." I've seen tfjs run simple RL game agents in-browser, too. The models are re-trained, tweaked, or slightly augmented - but not too much, since you only have 24 hours.

Reply on news.ycombinator.com