Environmental Understanding for Augmented Reality


A complex dance

When describing how computers understand their environment in an Augmented Reality context, it’s important to understand that there is a delicate balance between many different systems, all working together, in a constant feedback loop. Mapping, Tracking and Loop-Closure are the pieces in that system. Taken together these pieces form the problem set of Simultaneous Localization and Mapping (SLAM). The simplified flowchart below should help when thinking about the process:


Mapping the environment

In order to create an illusion of a virtual object that looks as real as a physical object, any AR system needs to understand it’s surroundings in a very precise way.

A real-time, highly accurate, 3D scanned map of the user’s environment is an ideal foundation for environmental understanding.

Sidenote: It is unclear whether sub-millimeter resolution mapping will be necessary for consumers to adopt AR – but we doubt it. We estimate that inch-level resolution resolution will be acceptable for the overwhelming majority of consumer applications. This is an important distinction when it comes to evaluating what kind of of scanning process to build for an AR system.

Centimeter accurate scan reconstruction from Pair system
Centimeter accurate single camera mobile phone scan & 3D reconstruction from Pair

Methods for mapping include:

  • Passive image processing, which evaluates RGB images only
  • Active scanning, which emits RF from hardware and interprets the “reflections” or response from the environment
  • A combination of the two

The look and feel of the device, as well as consumer safety are major considerations when determining what mix is required between active and passive mapping systems.

We at Pair are dedicated to purely passive systems. We think that passive mapping, tracking and localization allows for simpler consumer deployment, scales better with existing hardware profiles and is more robust in it’s failure modes.

The simplest active mapping system is the integration of a small form Infrared (IR) depth sensor, which consists of an IR emitter and an IR camera. Consumer hardware such as the Kinect and Structure sensor or platforms such as the Google Tango or the Intel Realsense, are the most readily available IR depth systems on the market. The benefits of IR depth systems are arguably shrinking as passive systems get more robust with mapping and tracking, however one place that IR will always beat passive visual mapping and tracking is in dark environments, as passive systems need to be able to take quality images to work.

Intel realsense camera. Source: Intel
Intel realsense camera. Source: Intel

More advanced mapping methods use higher frequency emissions, such as ultrasonic transducers, as seen on small robots, or structured light systems such as used with handheld 3D scanners. These higher frequency emitters can provide very high resolution mapping in nearly any type of environment which makes them the most robust of the mapping implementations. That quality comes at the cost of a larger form factor, larger processing footprint and marginal but non-zero increase in risk to the user from emissions. The major advantage that we see with these high frequency active systems is that they can map behind, underneath and around objects if properly implemented, making better maps with less work.

Ultrasonic rangefinder. Source: Hackaday
Ultrasonic rangefinder. Source: Hackaday

Tracking (Localization)

Along with a virtual map of the environment, tracking (localizing) where the user is as they move around the environment is a critical component. The process of extracting that viewpoint is called pose estimation. The primary discussion around tracking is whether “Inside-Out” or “Outside-In” tracking is the best approach.

A pose estimate can be derived from the environmental mapping system as the user moves, but this data does not fall out of mapping by default and there are complex processing systems that are implemented to make the whole process work. This approach is considered Inside-Out, as all of the tracking data comes from inside the user’s system, not from external hardware doing time difference triangulation. Describing inside-out tracking and odometry much further than this would go beyond the scope of this post.

In contrast, Outside-In systems typically use infrared or in some cases low power lasers to track reflective surfaces on the user, like the lighthouse system with the Vive or more contained VR systems.

HTC VIVE Lighthouse system. Courtesy: HTC
VIVE Lighthouse system. Courtesy: Steam VR

However for AR there are significant logistical problems that come with this as soon as you step out of a controlled environment. Basically for “outside-in” tracking, the entire world needs to have these emitters for it to work everywhere and we don’t think that’s reasonable.

Needless to say, inside-out is much more difficult to do correctly, but has the advantage of working anywhere all the time and is the approach we take with Pair.

Relocalization aka “Loop Closure”

Ok, so we have a virtual map of the environment and we know where the viewer is within it. So far we aren’t seeing anything new in this system though, just a representation of the world around us. So now we want to add something virtual to the world so that we get some benefit out of this whole thing.

For simplicity sake, we wont describe how to build the virtual 3D world beyond saying that it is identical to how you would do it for any 3D video game. The previous two sections describe how you get the first person view within the 3D world correct, and this section will cover how we make sure it stays correct.

There is a certain “stickiness” to that virtual world that is necessary to build in order to keep the virtual map and the real environment “stuck” together so that objects seem real. If the virtual map is is not corrected according to the real world, then any virtual object rendered within the virtual map will always mismatch with the real world.

To solve this, a process called relocalization or “loop-closure” verifies that the virtual world matches the real one. Loop closure detects when the viewer returns to a previously visited place and corrects the position of the world fixing any mismatches between the virtual map and the real world.

Think of it like this: If you close your eyes and walk around your house, no matter where you stop and open your eyes, you will know where you are because you have been there before. The critical piece here is that places need to be seen by an AR system before we can interact with them virtually – which is a large subject of discussion in and of itself.

This process must be repeated constantly in order to glue these two worlds together. Here below you can see these elements all together exploded out into their component parts. Virtual objects on top, a scan of the world in the middle and the real world at the bottom:

Bottom: Real World; Middle: 3D Scan of the world; Top: Virtual objects
Bottom: Real World; Middle: 3D Scan of the world; Top: Virtual objects


In brief, any robust AR tracking system needs to do three things in order to work effectively: Create a virtual map of the real environment, track where the viewer is inside of the virtual map and keep those systems stuck together.



Our analysis of the Goldman Sachs Virtual and Augmented Reality report


Goldman Sachs recently released a 58-page research report on the Virtual and Augmented Reality market. We analyzed their reporting on AR, focusing on their highest level assumptions without getting into nitpicking detail.

Our Bottom Line

This report is a reasonable starting point if someone has no background on Virtual or Augmented Reality. Unfortunately, it glosses over or misses entirely some major factors which will impact the direction of the VR/AR market for the next decade. These factors include analysis of the phone and tablet-based AR software market, possible convergence between AR/VR hardware, potential impacts of corporate subsidies on AR HMD sales and more rapidly shrinking Bill of Materials for advanced AR HMDs.


Focused on HMD

This report, like all other reporting on Augmented and Virtual reality focuses on Head Mounted Display (HMD) hardware. As we discussed before, focusing only on HMDs misses the next decade worth of massively valuable use cases for Augmented Reality. The exclusion is especially odd as 21% of the events identified in “Exhibit 7” were about companies focused solely on phone/tablet-based AR:

3 out of the 14 events highlighted were for mobile AR companies
3 out of the 14 events highlighted were for mobile AR companies

Although the latest news about Lenovo building a Tango phone was probably too recent to make the report, there was no discussion whatsoever about Google Tango in the report, despite Qualcomm becoming an early partner last May and despite google putting a substantial effort into Tango specifically relating to AR.

Ignoring the mobile AR market misses potentially ten years worth of opportunity in a large proven market with real customers and value. Failing to include it on a report about the VR/AR market shortchanges readers and leaves them uninformed.

AR hmd Use Cases and pricing

We agree with the report that cost will be the chief driver of the use cases that dominate AR HMDs. We also agree that enterprise will likely lead AR HMD use cases until 2025. That said, by 2030 we fully expect AR HMDs to dominate computing across all use cases including communication, and entertainment.

We are skeptical about the BoM estimates for what they call “Integrated” HMDs given what we know of the composition of the Hololens, which is used as a baseline for AR HMDs. Somewhat counterintuitively, we have high confidence that costs will be considerably lower for more advanced AR HMDs. Advanced display systems that Pair has begun working with take less processing than the existing state of art, reducing total computing overhead considerably.

BOM est

We think that with sufficient investment ($20BN+), further advances in VRD’s, improvement in computer vision systems, and improvements in deep learning applications, a low cost, high quality AR HMD will be available to consumers before 2020.

VR and AR convergence

As an extension of use cases, the report mentions how VR/AR are on path to “[replace] phones and PC environments.” If true, the way this process happens has major implications on the market and would be valuable for investors and market makers to know. However, an even cursory analysis of how and in what mix that replacement happens between VR and AR is absent from the report.

We expect in the long run that hardware will converge and VR/AR will be two contexts or “modes” on the same device. This view is not unique, however it is the subject of much debate from technical, business, and social perspectives.

The alternative conclusion which could be drawn from this report, is one in which a user would switch between two or even three separate head mounted devices: one for communication and information, replacing smartphones; one for work, replacing PC’s; and one for entertainment, replacing televisions and theaters. We don’t see that as likely in the long term.

Adoption Rates

AR HMD adoption forecasts almost always reference adoption rates for other consumer electronics devices, namely smartphones and personal computers. This report is no different in that respect, and like most others, it does not address one major economic function when comparing to smartphone adoption curves: Carrier subsidy.

As computing platforms go, smartphones are the only one to date that have predominantly been sold as a bundled commodity device, subsidized by mobile carriers. This is changing now, but in our view, if the majority of users had to pay for their smartphones the same way they buy a laptop or PC, adoption rates would likely be much shallower.


We fully expect that the AR HMD will overtake the smartphone by mid-century, but it’s unclear whether mobile carriers will subsidize these devices like they did smartphones. If carriers or producers do not find ways to structure costs for consumers, we expect that AR HMD adoption will more closely resemble PC’s rather than smartphones or tablets.

PC adoption


If this report is someone’s first real look at the VR and AR market we conclude that it’s a reasonable starting point given the absence of other comprehensive analyses of the market. That said, specifically as relates to AR, the report falls short in its assumptions and analysis in a few key areas which will be vital for the VR/AR industry over the next decade. Finally, readers are significantly disadvantaged by the lack of inclusion of phone and tablet based AR software and its share of market through 2025.


Promoting the mobile Augmented Reality applications of today will encourage adoption of smartglasses tomorrow


Executive Summary: The more consumers use the phone and tablet based Augmented Reality (AR) applications available today, the more enthusiastic they will be about adopting smartglasses when they become available in the future. If you are excited about smartglasses and Head Mounted Display based AR, you should be encouraging phone and tablet based AR today.

Comparing reporting on Augmented Reality technologies

Augmented Reality has been a hot topic in the technology press over the last year, due in large part to some massive investments made over the last 18 months. Here are just three recent articles:

The Business Case For Augmented Reality

The Value in Virtual and Augmented Reality

Augmented Reality moves forward with Investments, Products

The vast majority of the reporting about AR, however, is on companies and technologies focused on AR hardware, specifically smartglasses, otherwise known as head mounted displays (HMD). In fact, if you look at AR investments over the last two years, the vast majority have been put into AR hardware, not AR software.

Coverage of AR companies that are not focused on hardware is so rare that Matt Burns of TechCrunch introduced Marxent as, “an AR company that does not make AR Hardware,” when interviewing the Marxent CEO. The notable exception to this AR software coverage dearth was Apple’s acquisition of Metaio last year. Even then, its notability among reporters was almost strictly because it was Apple making the acquisition, and coverage focused largely on questioning Apple’s plans for AR hardware. Outside of the AR development community, the Metaio acquisition made hardly a ripple, but it was the most impactful event for AR developers this year, and profoundly shook the AR development community.

Mobile AR provides widespread value today

I will admit, I am biased. Our team at Pair builds Augmented Reality for phones and tablets, but that’s not because we don’t have the depth to do hardware. It’s because we want to reach the almost 2 Billion people worldwide that can realize the value of Augmented Reality using devices they already own.

Almost 2 Billion people can realize the value of Augmented Reality today with a device they already own

Let’s take a look at just a few examples of mobile AR that provide an overwhelming value to smartphone and tablet users:

  • Google translate has almost half a billion users. Ever used their mobile AR translation feature derived from their 2014 acquisition of Word Lens?

Image courtesy Endgadget
Image Courtesy Snapchat
  • On the educational side, Star Chart has well over 20 million users now and is the most intuitive way for amateur astronomers to accurately identify celestial bodies.

  • At Pair, we provide a faster and easier way for consumers to make decisions as they design their living and working spaces. Our users overwhelmingly agree that using our tool adds real value:

Image and reviewThese are just a few examples, proving that value added Augmented Reality applications are already available to users worldwide, and they aren’t just gimmicks. You simply can’t get these kinds of experiences through non AR applications.

Mobile AR helps us refine AR use cases

While the use cases for HMD AR are arguably much broader, more persistent and, in theory, more easily accessible than smartphone based AR, there remain a lot of questions about what the most successful user interfaces and experiences will look like.

Smartphone and Tablet based AR helps us as a community understand usage patterns, consumer needs, and best applications of AR. It allows us to gather data about how users purposefully interact with their environment, gives us hints on the best applications for AR, and helps consumers feel more comfortable interacting with the virtual layer we’re putting onto the world.

Mobile AR will not be overtaken by HMDs in the next 5 years

Just three years after they started outselling feature phones, and almost a decade after they were first introduced, Smartphones have almost fully penetrated the mobile market. Cheaper devices are outselling the more expensive handsets and entry level Android devices have handily won the platform war.

Even if we assume that the first generation of AR HMDs are perfect, they will necessarily be high end, niche products that are expensive, with a steep learning curve. If the release of the Oculus Rift pricing is any guide, the first few years of consumer AR HMDs will price out the vast majority of users. Given that the most promising AR HMDs are possibly four years away from a consumer release, it’s not a reach to say that smartphones will continue to dominate the mobile computing landscape into 2020.

Not only is consumer adaptation a factor in the growth of HMD adoption, but the infrastructure necessary for high accuracy, high bandwidth AR applications on HMDs will take time to build. The majority of the target consumers for AR HMDs however will still have their smartphones until that time.

The playing field is becoming more competitive, but there is plenty of room for growth in smartphone based AR. Developers, investors, and the press need to acknowledge the value smartphone based AR is bringing to a massive population already. Higher adoption of phone and tablet based AR capabilities will only drive HMD adoption faster once we get to the point that we can build and deliver quality consumer ready devices.



The Augmented Reality platform with the most content will dominate computing

Digital Content Visualized

In 1996 Bill Gates penned a prescient article titled “Content is King,” outlining why he thought the most successful internet based companies would be those that provided the most content, similar to broadcasters in a previous era:

Content is where I expect much of the real money will be made on the Internet, just as it was in broadcasting.

The television revolution that began half a century ago spawned a number of industries, including the manufacturing of TV sets, but the long-term winners were those who used the medium to deliver information and entertainment.

When it comes to an interactive network such as the Internet, the definition of “content” becomes very wide.

His predictions all came true and the biggest internet and now mobile based companies have significantly more content than their competitors.

Take the mobile music player market for example. When the Zune was released in 2006, the Zune marketplace had just over 1.5 Million songs available, largely inherited from the MSN Music network. Compare that with the 1 BILLION songs downloaded from iTunes by 2006. Ironically, a big part of why the iPod destroyed the Zune was that overwhelming content dominance that Bill had talked about a decade earlier.

Similar content dominance patterns hold with social networks like Facebook and Reddit, dating sites like OkCupid and Tinder, and with content publishers like Buzzfeed and Vice, just like Bill predicted.

Quality aside…

Before going further let’s distinguish between the amount of content on a platform, how much of that content is relevant to an individual user, and irrelevant content that users are shown. Finding too much irrelevant content or being bombarded with spam content, will push any individual user away from a platform, which is why platforms spend so much time trying to present relevant content to users.

It’s not good enough to just have the most content, that content has to match the needs of the different subsets of users. Platforms scale when they can give relevant content to a broad spectrum of users, while minimizing irrelevant or spam content. There can be too much of a good thing when that good thing is only one kind of content.

Content case study: Microsoft and Apple

There is a strong argument that the longevity of both Apple and Microsoft has resulted from their ability to leverage content effectively at the right time, leading the market alternately for the past 40 years.

Apple dominated during the 80’s and they did so in part by providing the best content for enterprise, VisiCalc being the key driver in the early days, and applications for home users later. Microsoft took the personal computing helm in the 90’s, delivered the best work content with the Office suite on Windows 3.1 and later the most ubiquitous of all content windows: Internet Explorer. Apple took that ground back as the computing paradigm was shifting to mobile, with the introduction of iTunes in 2003 leading to a 500% jump in iPod sales by the end of 2004.

iPod sales started doubling only two quarters later
iPod sales started doubling only two quarters later

Since then the heir apparent of the iPod, the iPhone, has pulled 50-65% of all Apple revenues since 2012, due in large part to it’s content ecosystem.

Apple Revenue by product
Apple Revenue by product

The next content arms race will be on an AR platform

Microsoft Hololens

Things are changing again however. Google’s Android OS has a much wider content network and now has dominant market share worldwide, if not necessarily in profits.

As smartphones reach market saturation, Microsoft is driving hard into Augmented Reality with the introduction of the Hololens. Apple’s acquisition of the then largest AR company in the world, Metaio signaled that it wants to be dominant in the AR market as well. While Microsoft has an early hardware advantage with the Hololens, their content pipeline plans are unknown and Apple can easily leverage its mobile infrastructure and history of execution to bring content to whatever new AR device it creates.

While these are the two big players, Google has the data, the platform, and the know-how to destroy both. With their recent interesting investments in the AR space, as well as history with Glass (which as we know isn’t AR, but is a good product proxy) they have already told the world they are going to be a contender in the AR vertical.

While Pair is not yet at the level of Microsoft, Apple or Google in terms of hardware, we feel strongly that the power of content is our force multiplier. A core focus of ours is giving users new and valuable interactions by providing the largest catalog of quality, relevant content in the AR ecosystem. Even though Pair has only been out a month and a half, we have the largest content catalog available on any AR platform, with over 400 products for users to try in their home or office. As we grow our content depth and scope, and the AR hardware market materializes and matures, we anticipate that others will follow suit.

A sampling of chairs from the Herman Miller collection in Pair
A sampling of chairs from the Herman Miller collection in Pair

As the AR landscape unfolds, we are keeping an eye on those companies that focus on bringing the most relevant content to the largest swaths of users in the easiest and most compelling way. Hardware will eventually find parity, but it’s the content delivery mechanism that creates, or sustains juggernauts.


3 ways Pair is transforming the design experience with Augmented Reality



When we set out to build Pair, our goal wasn’t just to make a fun and engaging way to view furniture and appliances in your home or office. We wanted to address the real problem of contextualizing what is often one of the biggest purchases a family or company will make. Visualizing how a new couch, refrigerator, or desk layout will look in place is actually pretty hard and most of the tools available don’t give a real understanding of how that new purchase fits with your décor.

Pair Room 1

With Pair, we’re already shaping how individuals and professionals plan their spaces, and transforming how design is done. We wanted to highlight the three major ways we bring all of that together.

1. Bringing the showroom to you

Yes, it’s our tag line, but it’s also what we really do. A showroom, after all, is a place that you can get an in-depth, 360 degree view of the products you want to try out. What better place to do that than the actual space that you’re designing for? It seems obvious now that we say it like that.


It’s not simple though. Our ridiculously smart engineers work tirelessly to teach your iPhone or iPad to understand the world around you in three dimensions. Here’s one of them:

Mike hard at work

By understanding the world around you, we make the design process simple, giving you the ability to simply drag and drop the products you need into your world.

2. Building the World’s largest Augmented Reality product catalog

We have over 400 products available today for you to design your world with. We are doubling that number every other week with the addition of new catalogs and categories of home and office goods. We even throw some fun stuff in there from time to time. Our goal is to have every product you could want, available to view in your space. From home and office furnishings and appliances, all the way to full size automobiles you can try in your driveway or garage, we want to be the one stop shop for anyone evaluating their next big purchase.


We respond to your suggestions, so tell us what brands and products you want us to put into the catalog.

3. Democratizing the design process

Traditionally, you needed to consult a design expert if you wanted to give your home or office a makeover. As with everything in the 21st century though, democratization of capabilities is the new normal. Pair gives everyone in the world the ability to design their space to the highest quality level.

That’s not where it ends though. If you are a designer and have your own layouts, or designs you’ve built in Sketchup, Revit, or 3DS Max, you can import up to 40MB of your designs every month, for free. Our upload interface is simple to use and supports a wide range of formats. Have a model that isn’t supported? Contact us and we can help get you up and running.

Pair Upload

The best part is that you can share those models with any Pair account.

What’s next

We’re just beginning to scratch the surface with the Pair design capabilities. We couldn’t be more excited to be leading the transformation of the design process and providing the world with incredible visualization tools.

See how Pair can transform your design capabilities: Download now or check us out on Twitter, Facebook or Instagram. Don’t forget to share the snapshots and videos of your designs @pair3d and tag them with #pairit.