Training AI to produce AR sky effects

How Archer trained an AI to produce magical AR sky effects

Dave Wallin

Manager of Innovations

"As an Innovation Labs challenge, we researched what it would take to create a Snapchat-like filter that replaces skies in images with rainbow skies."

The challenge

Platforms like Snapchat drive engagement by leveraging cutting edge AR (augmented reality) and AI technology to allow users to express themselves creatively. Snapchat’s Face Filters and World Lenses allow users to accomplish complex transformations that look fantastic and take very little effort. This year’s Hoagiefest concept involves magical rainbows in the sky – as a Innovation Labs challenge, we researched what it would take to create a Snapchat-like filter that replaces skies in images with rainbow skies.

How it works

To accomplish this, we trained an AI algorithm called a neural network to take a regular image and predict which parts of the image are sky and which are not. Neural networks can accomplish many image recognition tasks, but they require training examples. In this case, we provide the neural wetwork with an image of a scene, and a mask image that uses white to indicate which areas are covered by sky. By feeding it a number of example images and masks, we can train it to predict the sky mask for a given image.

An example training image and mask pair:

example training image and mask applied by AI


Even with a small training set of only 21 images, we were able to get good results. More training images would result in more accurate and robust predictions across a wider range of environments and lighting conditions.

Original image, predicted mask and result image with rainbow added in masked areas:

original image, predicted mask, and with rainbow affects added


Aside from Augmented Reality, image recognition can be used for things like spotting products in images posted to social networks, monitoring for inappropriate content, and automatically tagging and categorizing product images for online stores.

gif of rainbow mask applied

Once an algorithm like this is trained, it can be run in cloud, in a native app or even in the browser using tools like Tensorflow.js.


AI and AR make it easier for users to create video effects that previously only experienced designers or filmmakers could accomplish. Brands have the opportunity to give users new ways to empower users to create their own viral content. Some influencers have large followings on social media, so giving them the tools to express themselves while promoting brands they like can be a win-win situation.


Archer Quarterly
From Our Brains to Your Inbox

Get the latest news, insights, and industry analysis from The Archer Group.

Contact Us

We're a full-capability digital and brand agency that delivers on analytic, strategic, and content needs for a range of awesome clients. And we love what we do. Any other questions?

Your Name

Your Email

What can Archer do for you?