How Archer trained an AI to produce magical AR sky effects
"As an Innovation Labs challenge, we researched what it would take to create a Snapchat-like filter that replaces skies in images with rainbow skies."
Platforms like Snapchat drive engagement by leveraging cutting edge AR (augmented reality) and AI technology to allow users to express themselves creatively. Snapchat’s Face Filters and World Lenses allow users to accomplish complex transformations that look fantastic and take very little effort. This year’s Hoagiefest concept involves magical rainbows in the sky – as a Innovation Labs challenge, we researched what it would take to create a Snapchat-like filter that replaces skies in images with rainbow skies.
How it works
To accomplish this, we trained an AI algorithm called a neural network to take a regular image and predict which parts of the image are sky and which are not. Neural networks can accomplish many image recognition tasks, but they require training examples. In this case, we provide the neural wetwork with an image of a scene, and a mask image that uses white to indicate which areas are covered by sky. By feeding it a number of example images and masks, we can train it to predict the sky mask for a given image.
An example training image and mask pair:
Even with a small training set of only 21 images, we were able to get good results. More training images would result in more accurate and robust predictions across a wider range of environments and lighting conditions.
Original image, predicted mask and result image with rainbow added in masked areas:
Aside from Augmented Reality, image recognition can be used for things like spotting products in images posted to social networks, monitoring for inappropriate content, and automatically tagging and categorizing product images for online stores.
Once an algorithm like this is trained, it can be run in cloud, in a native app or even in the browser using tools like Tensorflow.js.
AI and AR make it easier for users to create video effects that previously only experienced designers or filmmakers could accomplish. Brands have the opportunity to give users new ways to empower users to create their own viral content. Some influencers have large followings on social media, so giving them the tools to express themselves while promoting brands they like can be a win-win situation.