Facebook’s 'Caffe2Go' AI Platform Can Transfer Video Effects in Real Time Using Your iOS Or Android Smartphone

Facebook started to slowly introduce its new deep learning platform called Caffe2go, which lets users capture and transfer video effects in real time using iOS or Android smartphones. Though the effects are pretty cool, the tech behind Caffe2Go is really interesting.

Deep learning requires content to “be sent off to data centers for processing on big-compute servers,” Facebook wrote, but with Caffe2go, the processing can be done from “the palm of your hand.”

The new platform is part of Facebook’s AI effort that includes the Lumos app used to take out images that violate its standards. Facebook isn’t the only company working on AI projects. Google released Tensorflow framework to the open source community and Microsoft made its Cognitive toolkit available to developers.

With Caffe2Go, AI has opened up new ways for people to express themselves. The new deep learning platform on mobile enables different possibilities. "We can create gesture-based controls, where the computer can see where you’re pointing and activate different styles or commands. We can recognize facial expressions and perform related actions, like putting a “yay” filter over your selfie when you smile,” Facebook wrote.

Similar to the Prisma app, Caffe2Go can transfer unique styles from artists like Van Gogh or Money onto a moving image. When it comes to processing live video it takes a well-equipped PC, but Facebook says they are “able to provide AI [processing] on some mobile phones at less than 1/20th of a second.”

Engineers had to design a software that can work with your smartphone’s memory and computing power. The team went out and created a UNIX-based system smaller than a similar program that works on Android and iOS. They also created a CPU feature called NEON that has the ability to improve mobile processing.

Facebook CTO Mike Schroepfer said at a Web Summit Tuesday, “It really is less about Facebook the mobile app today, and more about Facebook—the mission of using technology to connect people.”

Facebook is giving developers access to Caffe2Go via its stack, and is looking to open-source it “over the coming months.”