Blue Sky Paint app was just released on the App Store. I’ve been following its development for quite some time and indeed very excited to try it out.
What is Blue Sky Paint (previously Skywriter)?
Blue Sky Paint, or BS Paint for short (#bspaint) is a collaborative augmented reality drawing app where you draw things in the sky. With this app, the sky turns into a free social canvas where everyone can share their drawing creations for everyone else to see.
Blue Sky Paint persist all those sky drawings. The app requires a GPS, as your drawings will be saved in the place where you draw them. This means that when you relaunched the app and visit the same place the next, you’ll see your drawings. Everything you draw is publicly available for all other Blue Sky Paint users to see your creation Of course, because your drawings are made public, the developer warns for improper use.
I think other than the developer erasing improper content, there isn’t a lot can be done I think. I mean, you can’ monitor everyone’s work and the app doesn’t have any registration option either, so you can’t block users either. Anyway, for the first release, this isn’t needed but maybe something that will be needed in the future depends on how popular this app gets.
Let’s take a close look at the app in action.
The drawing experience is really fun. There is a color picker that is positioned near your waist and you can see it when you are looking down, kind of like a digital watercolor palette, at least this how it felt like.
One of the key features of this app is that supports occlusion, in this case, sky occlusion. Occlusion is one of the most needed features in augmented reality apps. It allows making virtual objects to be mixed with a real-world physical object. It usually works with the app scans your area and create a 3D digital replica of it, which can be used by the app. So for example, you can put a virtual object behind a real-world object and it will be occluded. Only if you physically move to the other side, you will be able to see it.
Now, I don’t know how exactly the developer made this “magic”, but the occlusion seems to be based on some sort of image recognition algorithm, which is based on the natural color of the sky. However the app does its magic, it works amazingly well By well, I mean, it’s super accurate. I draw things on trees and it actually could tell the difference between the tiny spaces where the sky is seen and the small leaves. Obviously, you can guess that if it does the masking so precisely, everything else masked super accurately. For example, I draw lines across buildings, structures, cars, road signs, and everything blended seamlessly with the environment. It’s also important to mention that this vision-based sky segmentation (as the developer calls it) feature was developed by the developers themselves. It’s not part of any currently available AR framework.
The app does have some hiccups. Considering that the mask probably uses some sort of a high pass filter that occluded everything other than what the app perceives as the sky, it did mistake some areas for the sky. I did try to trick the algorithm and try to see if it can be fooled when I point the camera to the floor, it didn’t work. I guess that part of the algorithm also takes the pitch angle into account.
Even with some misses, overall it works surprisingly well and I was very impressed by it. For me, this app was also a fantastic use case that shows the importance of occlusion for increasing immersion in augmented reality. As simple as this app is, the use of occlusion make is amazingly fun and entertaining. Even though I didn’t find other people’s creations, I cannot imagine how fun it would be going outside and see what other people have drawn in the sky, right outside my house.
It gets even better when being able to view other people’s creations in other places in the world or drawing with friends together in the same place collaboratively. By the way, when drawing with friends, you and your friends see each one’s drawing changes in real time on their device, a really cool feature that is only become obvious when using the app with other people.
I couldn’t ignore the fact that the app has very limited drawing functionality. Right now the app focuses on drawing simple color brush strokes, but obviously, there is many other opportunities here to add many types of embedded multimedia pieces, like animated GIFs, emojis, etc.
I like the effect when moving my finger over the palette, but creators will definitely need more than just that. This means that you are limited in the type of things you can draw with it. It’ like having a single burhs at your disposal and that is it. Still, I’ve seen the trailer and saw some really great drawings. The thick default brush is a good choice to start with because it makes each stroke stand out, but again, we definitely need more tools.
There also no option to erase what you’ve created. It might not make sense, because the idea is to have a canvas where everyone can write on. You also wouldn’t want some stranger to come across you drawing and ruin your work. Having said that, It would be nice having the option to have layers, so creators can save work and finish their creation later or be able to replicate their beautiful hard-worked creations to other places. The thing is, there is no user registration. There might be something in the backend that assigns some drawings to certain app unique generate Ids (probably to be able to ban misbehaving users, if it does exist), but as a user, I don’t get the feeling of belonging to that creation.
This leads me to talk about the social features.
One of my biggest disappointments is that this app, as for its current version, doesn’t have any integrated social features.
Imagine that there is registration and you can create your own account. This means that people who come by your work can see that you are the one who made that drawing, and follow your profile, even LIKE your drawing in the sky and share it with others.
Speaking of sharing, this app just shouts for built-in social features. Being able to see what other users are drawing, see some patches in the sky in other places around the world. Right now, from what I can tell, there isn’t anything like it. We need to have a Sky browser built-in the app, like maybe having Google Street integration as a browser or just a map where you can click and check out some of the creations. There is a problem achieving it though.
The thing is that in order to view the paintings in the sky in a given area, you need to have a viewing perspective as a user who stands on the ground and looks in the sky. The developer can just show you the art, but obviously it isn’t the same thing as viewing it from the ground, and of course, let’s not forget occlusion. The whole idea of this app is having the drawings occluded with the landscape environment. This means that other user can have a great AR experience only if they are physically in that location. The environment can change and of course, you can’t rely on historical scene view like Google Street View for displaying the content with occlusion.
I’ve seen this tweet by Kevin Habich (co-creator), who posted a view of the drawings as seen on a map app, which is only accessible to the developer. This gives the developers a bird’s-eye view of where people are painting. So I think this type of functionality might be nice. I think that the developer can use Google Street View (or a similar app) and place the content in a certain approximate area and occlude it with that view if it’s possible. I guess it is because this is just putting an image on top of a feed and have it occluded with some sort of a similar algorithm. Again, I am not a specialist in this topic, but something needs to be done to create a Sky Browser one way or another.
I actually imagine opening the app, and instead of just seeing the painting tool, I want to first see the social hub and discover new creations. Furthermore, might be not even less important, is the option to see if there are other creations nearby or even see if a creation is taking place in real time (if the user gives permission).
Having a profile means that you can meet people in your area and collaborate with them as well. Then when you create a new ‘sky painting’ together, it will be credited to both of you. Again, having the option to save your creation would be amazing, it can be seen in your profile, even if it’s overwritten later.
The developer an also allow users to add a short clip that shows their creation, and this clip can be viewed in the YouTube-like social area in the app. It would be great if the app can keep the history of the changes in the sky, so users can see the sky ho wit looks yesterday or a year ago, flick with the finger to browse through time!
As you can see, my mind goes crazy with ideas after using this app. This app really lay down some important fundamentals of how outdoor social apps in AR/MR should be.
Update: while writing this section, I had a chat with Aidan Wolf, one of the developers of the app and online friend in the AR community. He mentioned that the app uses GPS and compass at the moment He mentioned that the occlusion was created using graphics shaders and other computer vision related algorithms to make the magic work. The most important thing is that according to Aidan, the app is built with social features from the ground up, so we probably are looking at cooler social features later on.
Although the skies are obviously huge and are all around us, at one point, if an app like this becomes very popular, now or in the future, we won’t be able to see the sky anymore. So instead of painting in the beautiful sky, we will be painting other people’s paintings. It seems hard to just make a global erase at a certain point in time, like every month or so, because the experience needs to be continuous and obviously people won’t start drawing before an expected erase, let alone erasing the entire global creations that people have done. Maybe giving users the option to see the clear sky when they make their drawing is a nice option, so at least you can get back that feeling that you are drawing on blue sky again, which is one of the reasons I enjoyed this app in the first place and why its name is part of it (“Blue Sky Paint”).
Other things worth mentioning
I’ve noticed that there is some sort of progressive loading. For example, I made a drawing in one place, ent like 50 meters away. When I came back, that patch in the sky where I’ve originally drawn onto was blank. I start drawing back again, and only like 20 seconds after I start drawing, those drawings that I previously have drawn appeared again. This probably was done to reduce overload so, but I’m not sure.
I noticed that when drawing something in a specific place, I can see it not in its original location when moving to another place. For example, I draw something in a certain patch in the sky, when I moved like 20 meters to the right, I saw that drawing in front of me, although I draw it on a different horizontal plane. I kindly asked the developer to explain this, and he told me that they have created a feature that can re-project drawing to work with different perspectives, so drawing will become distorted but never appear s flat.
This is important when you deal with flat art, not 3D virtual objects. This means that without this feature if you viewed the drawing from the side, it would appear like a very thin line. This way, you can enjoy the beautiful drawings around you without needing to walk a lot just to get a different perspective, it’s like the drawings are facing you, rather than you need to face them in the correct angle.
This app obviously could have been even better with AR cloud, and ability to not just to occlude on sky but on other objects in the environments. I’ve seen many GPS-based apps that weren’t enjoyable because of the GPS accuracy, like with Atlantis AR World.. However, because the drawing is done in a large area, and the occlusion works accurately regardless of where the drawing is located using graphics shader, it maintains a great AR experience.
Othe type of AR apps that require positioned of virtual content in a high degree of accuracy will require AR Cloud technology, and can’t rely on GPS for that. This is why it was a smart idea to make such a GPS-enabled app that uses a big area where an accurate positioning isn’t a necessity than try to make it work in the entire environment.
This app with AR Glasses & Beyond
When you experience this type of app like Blue Sky Paint, you cannot ignore the fact that this app is a perfect match for AR glasses. I mean, picture this. You wake up in the morning, go outside, look and the sky and see all those great drawings in the sky, changing every day.
Now that your hands are free from holding a mobile device, you can use a controller or even your fingers to draw on the sky.
Having an AR headset will make the viewing experience feel more natural and immersive because you’ll see a unified vision of the sky when you walk or turn your head around. The virtual drawings in the sky then become a more permanent part of your conceptual perception of the space around you. It’s like wearing sunglasses, but instead of just seeing things better to UV filtering, it puts you into a dynamically ever-changing space.
Imagine people even not needing to be in a specific area to showcase their drawing, even working on their art from their computer, but positioning their art in a specific place and letting others see how it is drawn in real time, our sky would never be the same, they will change right in front of you.
Imagine using this type of social app with AR cloud. Imagine you want to write something on the floor for people who are currently at Plaça de Catalunya square in Barcelona, Spain. You are doing it from your device in a different country, other people are seeing it in real time using their AR glasses. Not just that, it’s dynamically changing every time. Because it uses AR Cloud, you can paint on any object, outdoors or indoors. Of course, we’ll need some sort of filtering to make sure we are not talking int a spam swamp but no doubt that this will be amazing to experience that. I am also not talking just about drawings, but images, videos, animated GIFs, live streams, 3D virtual objects, visual effects, social profiles, many layers of alternative realities, overlayed historical images of specific places, games, etc.
Yes, Blue Sky Paint gives us a glimpse of what’s possible. It inspired me to think of great ideas and actually get a look at the future. This little app, as small and simple as it is, is an important milestone. I’m sure there are many creative minds out there that sees this app and get inspired, and I mean, seriously inspired.
I’m Getting Emotional
I don’t do this in any review but I think it’s important. I’ve got really emotional when I used this app. The only thing that was missing was music, but I put music in the background myself.
So there I was, in the street, drawing in the sky while music is playing in the background. I’m sure if I was in Paris right now, listening to an atmospheric soundtrack I would be in heaven.
I felt like I am in another realm. Although I can give many ideas that help improve the experience, what this little app did, no other AR app did before it. I love when an app is able to make such an emotional impact. When I discover an app that really touches your heart, makes you imagine, feel and create.
Some may look at this app and say that it’s just a simple boring drawing app, but until you try it yourself, you can’t fully understand its impact. It paints the future of AR in glorifying vivid colors, shows us that the AR technology has the capability affecting our emotional state in a very positive way. It can excite, stimulate, encourage taking action and promote creativity as well.
These are some of those things that I pay attention to when reviewing apps. At that point, the only thing you want to do is just taking the developers who made it and I think I’ve done this by writing this in-depth review and sharing a full-through experience. I still think that I have a lot to discover and as I said, I keep this app with me for wherever I go. I can’t recall an app that triggered such a strong urge for exploration. The more popular this app gets, the better the experience will be.
Until those social features are added, you can share your creations on social media with #bspaint hashtag so I and other users can discover them. I really want to see what other peoples are painting, so please do share.
If I can describe the “Blue Sky Paint” app in one word, in an emotional and immersion standpoint, it would be “Breathtaking!” with a big exclamation mark; especially when you experience it for the first time. It’s the “Microsoft Paint” application for AR, an AR app with Michelangelo soul in it. Reaching for the sky takes a whole new meaning with this app.
The occlusion makes whole the difference in the world for this app. Without it, I wouldn’t even consider using it. The key elements here is immersion. The app is more exciting on the explanatory level than it a fully-pledged social drawing app in its current state, which it isn’t. The other thing is, of course, the real-time collaborative drawing experience (you see what other people are drawing in real time) and persisting the sky artworks. I’m telling you, if I wake up tomorrow and see that someone draw something where I’m at, you don’t know how thrilled I would be, I just know it.
Yes, it is a simple app with limited features. However, we cannot in any way ignore the innovation and novel app design that eventually help deliver such one of a kind AR experience.
In many ways, because I am aware of its potential, it doesn’t feel far from being complete. However, it’s simplicity is also something to be treasured and appreciated. This way, it can reach a broader audience. I will also judge it as such. I’ve seen many complex apps with tons of features that made me stop using them after a few seconds, just because they were so full of features. There are ways to hide some complexities and I do think that this app desperately needs more advanced social features, especially ones that make it easier for others to discover other people’s creations, not just on-location.
One of the greatest features of this app is the ability to discover other user’s drawings in the sky. For that to happen, the app needs to be simple and accessible. So in that aspect, I don’t see the lack of many other features, a big issue. In fact, on the contrary, I do believe that it was smart releasing the app with a small feature set. I want people to at least try the app, even for one minute, draw something in the sky and continue what they are doing, only for that little drawing to be discovered later. They can definitely do this with an app that doesn’t have a stiff learning curve. With a simple app, the chances that people will at least give it a shot and paint something in the sky is higher like that, so does the chance of other users to feel that discovery feeling when founding out that someone else was in this place and left a drawing for others to discover.
As these type of occlusion will become a common thing in many frameworks, users will pass the excitement phase and will demand more, much more. I mean, we already have the Magic Leap One headset that supports full occlusion of your surrounding environment (not sky occlusion). You can satisfy users with just painting on meshed surfaces, same here. The developer starts really well, but this app will need more updates and I’m positive those will come.
The app no doubt will have a great impact on people who tried this type of experience for the first time, and I’m sure for many it is because it’s the first time Sky occlusion is done in AR (correct me if I am wrong). However, the work doesn’t stop here and I hope that the developer has goals to take this idea to the next step.
When I was in Seoul, I remember trying out an app called Skrite. An app that let you put messages in the sky in the form of animated GIFs, text, etc. What I liked about that app is that the developer used Google Street View to show how certain messages look in different places. Good for previewing shared drawings.
Even considering the huge potential of this app to become so much bigger than it is right now, we cannot ignore its importance, originality and how immersive it is. This type of app should be an inspiration for all developers. It shows us that how amazing AR technology is when combined with other computer vision innovative technologies. You have an idea in your head, and you create a proprietary technology solution to help you achieve that goal. It’s important because it teaches us, developers, an important lesson, to search for technological solutions to help us make our idea a reality, and if those don’t exist, create them ourselves.
This is why I admire Kevin Habich, Lucas Rizzotto, Vova Kurbatov and everyone else who was involved with this project. They have created something amazing, that its importance goes beyond what the app does and that’s the beauty of it.
In my opinion, the contribution of this app to the overall AR app landscape is undeniable. Many future apps will build on top of that idea. It’s not just having an idea, maybe other people thought about it before, but it’s making an actual active move to make that great idea a reality.
I also want to add that there is a big difference between seeing this app online and actually using it. So even if you find the drawing and the interaction not your cup of tea, I highly recommend download and trying it out.
I had a blast with this app. It will stay installed on my iPad and I can’t wait to try it out every single day that I”ve outside, living my mark, discovering people’s creations. I do hope to see more features coming. The developers have a great unpolished gem in their hand and with this type of app, only the sky is the limit.
P.S. I will share more about cool features I want to see in this app in later articles, there is just too much to write about, and I can’t share it all in this review, so stay tuned.
Oh, imagine having a “Paint Bucket Tool” like in Photoshop and you click in the sky and boom, the entire global sky shifts its color. Just kidding, but an interesting thought, isn’t it? Like the sky is a now a tremendous collaborative Photoshop layer, insane! BTW, I’ve also wondered about the accuracy of this occlusion algorithm if it works well in different lighting situation (it did in my testing) it’s low light limitation and it’s accuracy based on the camera’s sensor. Something I will want to check out later as well.
It would be nice also to have an option to add audio to enhance the experience, which triggered when you point your device at a specific artwork in the sky. It would be cool to have some sound effects when you brush like a windy brushing sound, it would make the drawing feeling more exciting in my opinion. Anyways, I’ll save the discussion of tons of ideas for the following article.
You can download Blue Sky Paint for iOS, free from the apps store here.
If you read the review until here, I have a little surprise for you. When using the app, try out those two gestures (pssst.. don’t tell anyone, it’s supposed to be a secret).
1. Two fingers down on the color wheel unlock shading for the current color
2. Pinch zoom gesture on the sky to scale the brush smaller or larger
Update September 23, 2018: I have decided to award Blue Sky Pain app AR Critic’s Editor’s Choice award for its innovative and creative app that delivered a breathtaking AR experience, one that I haven’t experienced before. It joins other few selected apps that were also been given this award – Congratulations!
I might be the first to give an award in USDZ file format, nonetheless, here we go (you need an iOS 12 and ARKit-enabled device for the AR to view this golden trophy with our logo on it)!