MLJ.jl: A Julia package for composable machine learning
alan-turing-institute.github.io
alan-turing-institute.github.io
That being said, MLJ tends to wrap a lot of models from scikit-learn with a composable wrapper, meaning you end up having to manage Python dependencies and you lose Julia features such as multi-threading (Edit: when using Python models, multi-threading is supported when using Julia models).
Long term I am sure MLJ will slowly replace scikit models but right now it can be little painful to use if the model you want is not implemented in Julia.
Did I misunderstand something?
Presumably, all fast languages are at a disadvantage, then.
This is really a minor issue stemming from the relative maturity of the languages - if Julia becomes more established I would hope usage of external libraries which don't offer a performance advantage (ie everything besides C and Fortran) eventually gets replaced with native packages to preserve the sanity of the users.
Keep in mind though, that this could be a hurdle to those who are less multilingual. So even though it's not a weakness to everyone, it is to many.
I think the two-language problem begins to be as we move forward a network of multi-languages calling each other instead of either having to rewrite everything from scratch either relying on a C/C++/Fortran/Rust lib and the C ABI to patch them. We may finish with a set of "meta"-langurust in C code nameages permitting the interaction of a large number of libraries and programming languages outside of their silo.
Moreover Julia coupled with Pycall could "corrode" à la Rust for a more smooth migration to pure Julia.
There is no way to bootstrap a scientific language without leveraging existing ecosystems. But this is a starting point. Whenever someone find a sklearn model to slow, or misses the automatic parallelism that Julia brings, they can just implement the model in Julia, and bit by bit the non-Julia bits get swapped out where it brings a real benefit.
And not instantaneously. Developer time being finite, using what works from existing libraries while waiting for a pure-julia implementation seems like an excellent solution.
Julia hides Python, Julia good
I understand that using Python feels wrong, because Python is not know for its performance, but still it seems like a Practical decision, rebuilding everything from scratch will be too much effort
The python interop is just a stopgap
I think you can only make the snarky remark because you don't truly get the 2-language problem. The problem has zero to do with the complexity of dealing with dependencies, as if the hard problem of scientific computation languages is like a Linux distribution with too much bloat. No, no, no. It's the cognitive load to solve scientific problems, when at the end of the day performance really matters. We need a language that is as terse as mathematics where we can customize all the way down to a single array entry, without performance loss. Python itself will never have the necessary performance. To do truly new numerical stuff you cannot avoid the computational kernel details (e.g., PDEs or image processing). So you're forced, as a researcher, to become good enough to be effective at two languages, and you often have a complex data exchange interface between the two, and it sits right near the critical conceptual problems that you're trying to solve.
The problem I ran into is that it can be very unclear when things don't plugin exactly as planned. You get a typing error that looks like C++ template errors. Or your data doesn't work quite right with a model and needs some transforming but the documentation doesn't spell how to do with your situation. There aren't nearly as many StackOverflow questions on using Julia as there are Python ones so you can't just look up a very specific question and get answers.
I’d love to train a ML algorithm to do this, and I’ve been building up the before and after pictures over time using my manually cleaned up customer fingerprint images. Can anyone give me suggestions or pointers on the sort of algorithm that may be best suited for this task? Just something to get me started down the correct path would be very helpful.
Step 1: write out the steps you do to manually clean up.
Step 2: Depending on: (1) how many samples you have, (2) what is defined as "before" and "after", (3) what performance level is acceptable, there are many algorithms to choose from. Transform learning may be the target, reinforcement learning or CNN as well. It really depends on Step (1) and (2.1).
If so, you may be able to skip ML altogether and use a deterministic software program to classify light versus dark areas (what you're doing with your pencil). ML might help with mapping smudges to the right shape/sizing.
That said, if you have a ton of examples, then yeah an image recognition task might help.
If you can automate it using a combination of Gimp filters + Inkscape, you probably can do it pretty easily convert that process into code.
[1] https://docs.opencv.org/master/d2/d0b/tutorial_periodic_nois...
GAN's are hard to train for a new ML practitioner, VAE can have blurry results if you are not using skip-connections like in U-NET.
It sounds like you have training examples (images pre and post processing), but would benefit from an approach using transfer learning since you don't have that many.
Good luck!
The focus of this package is on the "plubming" of auto-ml solutions. The work of designing solutions based on multiple discovered models is rather unsupported in the current state of the art, and this package looks quite supportive of it.
A big contribution (if it works in practice) is the idea of a scientific type describing what data is and how it should be mapped to algorithms - a systematic way of doing this is the underpinning of a process for model selection.
If anyone has a good introduction to recommend which is clearer than the official docs, I'd definitely appreciate it.