Over 25 percent of the 500 companies surveyed by IDC are testing augmented reality (AR) prototypes. Much of this interest, however, is stuck in the prototype stage because most hardware platforms are prototypes and many of the tools needed to build the killer app to move into production are still under development.
Vuforia adds an important piece to the AR toolchain puzzle, though, with its announcement today of Smart Terrain for Google’s Tango tablet.
Smart Terrain, announced in conjunction with Google and Unity3D during the opening keynote at the Vision VR/ AR Summit, builds on two mature components of AR: Vuforia’s early experience solving a very specific AR development problem and Google Tango, which looks like the first AR platform ready to emerge from prototype status and become a mainstream platform.
Vuforia started working on solving the problem of how to create content that interacts with the AR device’s sensors to augment the reality in the user’s field of view with dynamic content. One of the AR apps most familiar to consumers is Wayfair’s interior design app that lets users overlay rooms in reality with virtual designs, dropping virtual objects such as furniture into the field of view to see an accurate and realistic rendering of the object in the room. Vuforia builds on this concept, with one significant difference, the objects can be dynamic. The best way to explain the capability is with this short 1-minute, 33-second video by Vuforia.
Though the demo leaves the impression that Smart Terrain is a game development tool, it could be used for enterprise applications such as step-by-step technician instructions.
Developers can build Star Wars Holochess that interacts with the environment
Dynamic content, effects and animated objects can be developed and added to the field of view that accurately interacts with the objects and planes in reality. The Star Wars Holochess game created by animating digitally rendered physical models for the movies a New Hope and the Force Awakens (seen in the 32-second video below) can now be built with Vuforia’s software developer kit (SDK).
A Holochess-like game was built by Star Wars animator Phil Tippett using Vuforia’s old SDK. The difference is today, the characters’ animation will not be limited to the table, but can move throughout the 3D space of the room and interact with planes, walls and other physical objects.
How Smart Terrain works
The problem of overlaying dynamic objects into reality is not new to Vuforia. The engineering team started working on the problem as a Qualcomm business unit a couple years ago. They were building conceptually the same features into Smart Terrain based on the PrimeSense’s depth sensing and acquisition chip. But Apple acquired PrimeSense, locking up the depth-sensing technology and posing an obstacle to the Vuforia’s project.
After PTC acquired Vuforia, the Qualcomm strategy pivoted from building software and components that run on every mobile device to building enabling software for other companies’ platforms. The Tango platform represented by the Asus Zenfone AR and the Lenovo Phab 2 is the right entry point because the hardware is in its third design iteration.
Tango’s sensors provide a high-fidelity point cloud, relative x, y and z coordinates of the physical space in the field of view. Vuforia’s Smart Terrain SDK computes a scene graph from the point cloud in real time. The scene graph is a hierarchy of real and virtual objects. In the video above, the tabletop is the parent node, the playing card is a child node, and the animated characters, effects and physical interactions with other objects are another child node.
Content such as the animation in the video demo is launched when the SDK has been programmed to recognize something in the field of view, such as a pattern on a flat surface, the geometry of a scanned object or a Vumark (which is any graphic the developer designs to trigger the content launch). Vuforia chose conventional computer vision techniques for its object recognition because it is mature and works well with the Tango time of flight sensors to create a high-fidelity and accurate perspective of reality.
Interaction with the field of view rendered by the SDK has 6 degrees of freedom (6DoF). 6DoF translates the field of view along the three perpendicular axes to match the change of orientation of the Tango device held by the user. For example, if an object like the helicopter in the video demo flew up and out of the field of view, tilting the Tango device upward and looking through it would regain the view of the object.
Smart Terrain, a developer’s perspective
Developers have a few implementation choices.
The Smart Terrain is integrated into the lingua franca of AR and VR development, the Unity3D AR and VR authoring platform. Unit3D will provide cross-platform compatibility for Android, the Hololens and iOS/ Xcode.
The SDK also address a big hole in the AR toolchain. Developers have two choices when building AR apps. Either they build an app with Unity3D, which are very high-level authoring tools, or they build a native app using C++ and graphics libraries such as OpenGL or Vulkan. The later is very difficult because such skills are rare in the developer community.
Although authoring tools, such as Unity3D, are very comprehensive development tools, sometimes developers have a problem that has not been envisioned by the tool makers and for a variety of reasons, cannot use a highly abstracted tool authoring tool. Integration of the SDK into Android Studio for Tango devices and rumored Tango headsets, Microsoft Visual Studio for the Hololens and iOS Xcode simplify the development of these apps because developers can build using familiar languages, such as Java in the case of Android, C## for the Hololens and Swift for iOS.
The 40,000 apps built with Vuforia’s library and about the same number of apps in development means developer familiarity will help developers adopt the new Tango-based capability.
Vuforia President Jay Wright told me they will release a Microsoft Hololens version. Though he stressed cross-platform support, he did not mention a target date for an iOS AR device.