Using AI and AR to improve ArcellorMittal's production process at a Hackathon

I'm with you on this. I can't imagine any other complex discipline where experts offer their time & creativity to the lowest bidder so quickly.

Programming seems uniquely generous in this regard. Or stupid.

Thanks for your view. However, it is important to see this event also for what it is. It is a way for Arcelor to think out-of-the-box, and get input from people that are not directly linked to their business. Secondly, it is a way for students to get a glimps of what the corporate projects and challenges are.

Additionally, for us, as professionals, it is a way to show, although in a very rough fashion, what is possible, and where we can help. We saw this event more as a team building than work.

The result we got at the hackathon is a very rough prototype, not a finished product. In no way, this can directly be implemented on the floor. As such, input from experts is still required, either us, or other experts in the field.

Therefore, these hackathons are useful, in my opinion, not only to the company, but also the participants :)

I agree. While I understand the OP's point that it's a valuable experience for the team, a glimpse into solving real problems, it's not a finished product, etc., I still think ArcelorMittal captures the bulk of the value here.

They had 70 teams competing. The team in the article had 5 members working for 36 hours, with 3 hours of sleep. Let's say the team members would fetch $150k/year. $75/hr. 5 people, 33 hours. $12,375. And there were 70 teams competing for $10k + snacks.

On the flip side, how much does ArcelorMittal lose on a single failed weld? I would venture a guess it's well north of $10k.

For you, and I'd say the whole HN audience, it is quite obvious that spending money in an AI approach would be extremely useful. For a steel producer, this is not obvious at all, and consider it a long shot.

I would even bet that the idea of whoever organized the hackaton was not to get free labor, but to prove to someone up there that AI can bring new interesting solutions. In view of the results, I'm quite sure they will be spending more in the future, maybe even starting a new department or a serious research program or something like that in the long term.

Why are you bothered by it? The 70 teams that joined the hackathon did it out of their own free will, and probably enjoyed it?

Great hack. Really enjoyed reading this! Noob question: the weld as shown in the AR video looks really detailed (fissures, deformations etc). Was that rendered by you using the heightmap data alone or did you have access to top and bottom images of the welds as part of the training data?

I suggest changing "conjest" to "condense".

6 months, here's why: They random forested height data, everyone random forests these days. It's quick and dirty and yields good accuracy. Not only that, everyone seems to support a built in Random Forest training model, programs from pandas/numpy/scikit to R has it built in. Height data seems the easiest to go by, to short and it has a kink that may cause cracking in the future as the metal oxidizes. What were their 3 models? Who knows? But their idea of combining 3 models, that 3 different teams have made is standard in all DataScience competitions. This is called ensembling. ArToolkit for Unreal Engine, someone else made that and they probably just connected a bunch of blocks together with Unreal's script engine. Honestly, 6 months max and you can do this in a day yourself.

I am no expert into AI. The AR part we learned because we are busy in VR. It is a natural evolution in that case, as we can use the workflow we are used to in VR.

The AI part however is more complex, as it requires understanding of the concepts and methodologies behind it. We used Deep Learning methodologies for this project. This is an upcoming field in AI, which evolves very quickly. If you are interested in it, it's a great time to start learning the basics of it!

Indeed, thanks for the comment! Unreal Engine is a great engine to quickly build prototypes in. We are used to work with the included VR Template of Unreal, and it most of the times works great.

Also for more serious projects, Unreal did not let us down.

Hey ! I am one of the AI guy of the team. I agree that if the solution gets implemented on the lines, they will probably make more net money out of it than what was spent for the challenge (which is more than 8.5k€ and some pizzas as said in other comments BTW, the challenge was probably more expensive to organize). However, let's be realistic, they will have to put a lot more money into this before they get a viable product to insert in their production lines. The solution we have built is way too simple regarding actual operational constraints (e.g.: operator safety).

Hi! I'm from the AI subteam. We actually didn't have time to implement this part of the solution. Our idea was to use this library: https://github.com/raghakot/keras-vis

Thanks! Good question, the AI part is clear, it will indicate if there is a big chance of break. The AR will help the operator decide without going to the actual break whether the AI is correct.

The AI only suggests, does not decide on its own. As such, an operator will be able to see the break in full detail (the top and bottom and up close) to better be able to decide whether to continue production, or do a reweld.

Also, sometimes the AI will not be able to provide a good input, as new materials might be welded together and no data is available for them. In that case, again it's up to the operator to actually give the go or no-go.

The AI will inform the operator after about 2 seconds, the operator then still has about 8 seconds to finalize his decision.

The current prototype is not really usable, as it has some big usability issues. We might work further on improving this prototype with the organizer, depending on their interest.

FTA: "Furthermore, we decided that we would require a setup that actually worked. Given that we did not have any AR hardware available (eg AR glasses such as a Microsoft Hololens), we quickly built one ourselves using a production helmet and a webcam."

Medium images sometimes take a while to load, but if you wait you can see their "webcam on a hard-hat helmet" setup for the demonstration to the judges. The overlaid AR + video feed then shows up on a monitor.

They did not have VR glasses, so they simulated them using a helmet-mounted webcam and a monitor. The AR weld risk profile (my words) is overlaid on the captured video images. See the article's video with caption 'Screen-capture of the AR experience...'

Nowhere, it wasn't actually AR.

It was a video feed that had the 3D model overlayed on the desktop/laptop the video was feeding from.

This isn't just semantics - moving from their setup to an actual AR HMD - even if it was just the current state of the art (Hololens, ODG, Epson etc...) wouldn't be trivial and would likely be a different workflow.

We used two convolutional neural networks, one for each side of the weld. Features extracted by these nets are then concatenated, as well as the meta data. Two fully connected layers are then applied on this vector, giving the final prediction.

Everything was trained on the raw images resized to 20x150 pixels.