Life after Kinect: PrimeSense's plans for a post-Microsoft future

When PrimeSense founder Aviad Maizels put a prototype of a 3D sensor on a chip in front of Microsoft in 2006, he had no idea it would lead to the biggest turning point in the Israeli startup's history. Four years later, its partnership with the Redmond giant resulted in Kinect, the motion-sensing camera that made headlines around the world. In 2013, however, Microsoft unveiled an all-new Kinect, the result of years of entirely in-house development -- without PrimeSense's assistance. As fate would have it, the company returned to its chip-making origins a year ago, creating a new product called Capri, a cheaper, lower-power and tinier version of its 3D system-on-a-chip; so tiny, in fact, that it's designed to be embedded inside tablets, laptops, thin displays and smartphones. With 3D use cases that go far beyond Dance Central, the Capri is the latest sign that PrimeSense is ready to break free from its video game roots.

My first meeting with Maizels was at a hotel suite in the Renaissance Las Vegas during CES 2013, where PrimeSense first unveiled the Capri sensor to the world. Our second face-to-face interview was decidedly less glamorous. It took place at a Peet's Coffee in Terminal 3 of SFO at 7:30 PM, hours before he was to board a flight back to Tel Aviv. It was the only available time slot in his busy schedule. When we met, he was dressed in an all-black ensemble consisting of a long-sleeved pullover and fitted trousers. He appeared casual, but put-together; his close-shaven chin and close-cropped hair still exuding a professional demeanor. Maizels had just arrived at the airport from Silicon Valley, where he spent the previous few days in a series of meetings. When asked who the conferences were with, he declined to say. "We're working hard on deals," he said slyly, with a tired smile and a strong Israeli accent. Tall and slender, Maizels is soft-spoken, yet articulate in a language that clearly isn't his first.

As exhausted as he was, at least he wasn't the bundle of nerves he was seven years ago when he and a few fellow co-founders brought an early prototype of their 3D sensor to the 2006 Game Developers Conference in search of a business partner. It was their first time at the famed video game trade show, and they could only afford what Maizels called a "garbage hotel." In what can only be described as a stroke of luck, they managed to impress enough industry players to score a meeting with Microsoft at E3 in May of that year.

That meeting was the catalyst for Project Natal, which Microsoft introduced at E3 2009 as a teaser for what would eventually become the Kinect. It was PrimeSense's first major partnership, and one that placed it on the map. Growth was inevitable, and it now has three offices in Asia, two in the US and one in its home base of Tel Aviv.

It's a far cry from the company's humble beginnings in 2005, when Maizels and his fellow co-founders -- Ophir Sharon, Alex Shpunt, Dima Rais and Tamir Berliner -- found themselves out of jobs after mandatory stints in the Israeli army. Maizels met Sharon, Berliner and Rais while working in research and development for the military, while Shpunt was a friend of a friend. Everyone came from science-heavy backgrounds, with degrees in engineering, computer science and mathematics scattered among them. Seeking to carve out their own path instead of just getting a job, Maizels gathered the group to, as he put it, "come up with the next big thing."

PrimeSense co-founders [left to right]: Alexander Shpunt, Dima Rais, Ophir Sharon, Tamir Berliner and Aviad Maizels.

What Maizels and his colleagues decided upon was a "technology that made technology itself disappear." Berliner explained that even today, technology still exists as an interaction between man and machine. "When we're playing games, you need to do it yourself. You need to actually use the mouse and the keyboard ... the machine doesn't understand what you want; it does what it's told."

Berliner explained the philosophy with a childhood anecdote: "When I was 10 years old, my kid brother was 5, and he wanted to watch an episode of Teenage Mutant Ninja Turtles. The episode had a very specific name, and he told me to write it on an empty video cassette. I asked him, 'Why would you want me to write it?' He said that if I write it on the tape, that is the show the machine will play. It seems really silly now, but that's how we want technology to work... we want it to be magic."

As most of the founders were gamers, video games became their starting point. "It seemed like the whole gaming industry was stagnant," Maizels said. "Finish one game, then you get into the next game, and it's a similar story, similar actions." The group let their imaginations run wild and wondered: what if they could wave their hands and have it be as if they were holding a sword? What if you never had to bump into walls ever again? What if, in a first-person shooter, you knew to duck and cover to dodge bullets instead of thumbing a d-pad?

"We had all these questions," Maizels said. "But we didn't have an easy answer."

The key question that emerged out of the initial discussion was what if a machine could see? That led to the idea of a device that could capture a person's movements with "sight," which the team pegged as the most important sense shared by both man and machine. And so the company's name, "PrimeSense," was born.

In late 2005, they cobbled together enough savings to rent a couple of conference rooms inside a Tel-Aviv office building. In early 2006, they hired Ziv Hendel, who helped out so much in the company's early days that Berliner considers him an honorary co-founder. The group breakdown was as follows: Sharon and Shpunt were in charge of hardware; Shpunt and Rais took care of algorithms and physics; software was in the hands of Hendel, Rais and Berliner; Hendel and Berliner took care of demonstrations; and Maizels ran the show, keeping the all-important business side of things afloat. Which, it turns out, was easier than Maizels thought.

"We were super lucky," he said, pointing out that the gaming industry isn't exactly the sort of business that venture capitalists in Israel invested in. "Most investments are in security, enterprise software, internet companies ... known entities with a proven record."

However, PrimeSense was located in Israel's equivalent of Silicon Valley, or Silicon Wadi as it's known, and he was able to find investors who believed in the product. "We got funding pretty fast ... I made my pitch that PrimeSense represented new paradigms of interactivity... and got early starting money to play around with." Early investors included Gemini Israel Ventures and Genesis Partners, both based in Herzliya. Each had backed successful startups in the past, including Diligent, a storage company that was eventually purchased by IBM, and Kidaro, a desktop-virtualization vendor that Microsoft acquired.

PrimeSense President and Founder Aviad Maizels [left] and CEO Inon Beracha.

The investors offered PrimeSense a bridge loan, which gave the company a lot of money up front, but also for a shorter period of time. This meant the group needed to show VCs constant product improvement to assure them their investment was well worth it. "When the investors visited, we have to show them progress," Berliner said. "What they saw last week, has to be nothing compared to what they see this week. On top of that, we also had to create demos that targeted the general public."

After months of trial and error, they came up with their 3D sensor on a chip. It acquires depth via "light coding" technology that processes a scene with near-IR light and then uses a CMOS image sensor to read the coded light back from the scene, using a variety of algorithms to extrapolate the 3D data. Shpunt was the one who came up with the initial design, which then evolved over time as the team wanted the product to be as affordable as possible without sacrificing its overall integrity.

"The technology has to be cheap," Berliner explained. "It needs to be cheap, and good enough. That doesn't sound like something you want to hear from a startup who wants to conquer the world, but we felt that was key to wide adoption."

The first hardware prototypes were pretty simple in terms of cost. "Some of the hardware was loaned to us by friends and colleagues," said Berliner. Some was purchased with the little money they had. The initial short-range prototypes were actually ready around September 2005 -- about a month after the company was formed -- and that was even before they got into an office. By December, they had their first "full-body" prototype -- meaning the 3D sensor was able to track an entire person from a few feet away. They were also able to whip up several demos using OpenGL 3D point-cloud visualization software that showed images dancing and playing piano in step with real-world movements. Other demos included game integration where they mapped the person to controls like skateboarding or driving a car.

The resulting product was a white plastic box that contained an RGB camera, an infrared sensor, a light source and that PrimeSense 3D-sensing chip. It was this device that made its way in front of Microsoft employees, including Alex Kipman, an incubation director for the Xbox 360.

"It was the best thing he'd seen in a long time," Maizels said. "Not just in the gaming market, but in general." Kipman, who hails from Natal in Brazil, named the project after his birthplace.

The next few months were a whirlwind of activity for the small Israeli startup. They expanded their operations, attended meeting after meeting with Microsoft and worked with the tech giant on implementing Redmond's own take on the software and hardware. One of their biggest challenges came in the form of Nintendo's Wii, which debuted in the US in November 2006.

"[The Wii] was close to what we're doing," Maizels recalls. "We were terrified. We thought, if it's a totally dumb experience, people will lose interest in it, and then no one will want to hear about our product. But if the Wii is great, maybe it's good enough and they don't want to hear about our product!" Looking back on it, however, Maizels thought the Wii was actually a great thing for PrimeSense. "People couldn't articulate holding the controller." He believed their product could bridge that gap. "We have a technology that can see, but cannot be seen ... The gaming industry was searching for something unique after the Wii."

As it turns out, the Kinect did prove to be incredibly successful. According to the Guinness World Records, it's the world's fastest-selling consumer electronics device, moving more than 10 million systems by March 2011. Microsoft sold an average of 133,333 units per day between its launch on November 4th, 2010 and January 3rd, 2011, alone. The Kinect was a hit.

But by the end of 2010, PrimeSense was already looking beyond Microsoft. It partnered with ASUS to develop the Xtion Pro, which uses the same 3D sensor tech from the original Kinect, but exclusively for PCs. Another collaboration was with Eedoo, a Beijing company, which made a Chinese-only multimedia entertainment console that tried to use a time-of-flight (ToF) camera, but then chose PrimeSense's "light-coding" tech for gesture controls. In November 2010, it partnered with Willow Garage, maker of robotics applications, and Side-Kick, a motion gaming startup, to found a non-profit organization called OpenNI, or Open Natural Interaction. The goal of the foundation is to promote and support 3D apps by offering low-level hardware support and visual-tracking software to the open-source community. To that end, PrimeSense offered up its NiTE Middleware as a free open-source download, so that developers could use it to create their own 3D applications.

OpenNI has already led to PrimeSense implementations far outside the video game realm. For example, France's CRIIF (Centre de Robotique Intégrée d-Île-de-France) has used it on its SAMI robot prototype so it can detect obstacles and avoid bumping into things; a retail solution called Shopperception used PrimeSense sensors to analyze consumer behavior in retail stores; and a startup known as Matterport utilizes the tech to scan and render any room into a pre-measured 3D floor plan for easier interior design and furniture shopping.

A family portrait of PrimeSense's products [left to right]: the Carmine 3D Sensor, an early pre-Kinect prototype and the Capri Embedded Reference Design sensor for compact devices.

"If you asked me back in 2005, 'Where do you see technology in seven years?' I would've said sensors would be everywhere," Berliner said. "My goal would be to make sure there's a sensor in every room in every house or office or mall. For example, when technology knows who you are and where you are, you won't need to ever enter your login and password ever again -- it should already know you." Maizels told us the sensors are already in use in fitness centers in Europe, and he hopes to roll them out in senior centers to make it easier for caretakers to help elderly tenants -- it would alert them if someone had fallen, for example.

Despite the early success of PrimeSense, however, it did suffer a bit of a setback in 2012, when the company laid off 50 of its 190 employees. Maizels explained it was attempting to change its product roadmap, to reinvent itself instead of "riding on the fumes of old technology." Which, it turns out, was a good move, seeing as Microsoft went with an in-house solution for the next-generation Kinect. Still, Tal Dagan, PrimeSense's VP of marketing, had nothing bad to say about its former partner.

"The fact that Microsoft chose to continue investing in 3D and brought an internal solution is a testament to their commitment to 3D and to the success of the Kinect to date," he said. "We still believe that PrimeSense has the best and most advanced technology and 3D solutions in the market."

To that end, the company took a bold step forward last year when it debuted the aforementioned Capri, which it claims is the world's smallest 3D sensor. PrimeSense boasts that it has three times the depth resolution compared to its predecessor, with dimensions that are 10 times smaller and with 50 times better ambient-light resistance (the ability to work in daylight).

"The Capri is the future of PrimeSense," Dagan said. "Not only is it smaller, lower-power and cheaper, it also has better depth, better middleware that can actually run on a mobile processor... our end goal is to make it small enough to make it into every consumer device."

An example Dagan gave was to use a Capri-equipped smartphone as something that can measure depth: "Instead of having to measure my daughter's height every two weeks, I could just take a picture of her with my smartphone, and it'll automatically know she's grown by a few inches based on her profile and previous height." He offered other potential ideas like a portable gaming device that utilizes motion gestures, a car-docked handset that sounds an alarm when you're nodding off to sleep mid-drive or simply the ability to scan an object for a 3D printer.

Dagan said the company is working hard on pitching the Capri to OEMs, hopefully to have it integrated in upcoming mobile products. When asked if there were any specific challenges to selling the sensor, Dagan deflected. "As in any new revolutionary product, the challenges are great. Our challenge as the technology provider is to bring to the OEM a technology that will allow these new revolutionary experiences while being robust and low cost." The company wouldn't reveal any potential partners either, but they did say the Capri was small and flexible enough to be embedded into "any consumer device available today."

More Info

• PrimeSense's OpenNI provides the best Kinect drivers yet, from someone who would know.

• ASUS Wavi Xtion motion sensing control system demoed at CES (video)

• PrimeSense demonstrates Capri 3D sensor on Nexus 10 (hands-on)

As we wrapped up our interview at the airport, I asked Maizels what his favorite Kinect game was. He said he liked playing Fable: The Journey, but instead of describing the game, he digressed. "Right now, it seems that there are two camps: people who are hardcore gamers, and people who play Kinect... But the experience should rule; there shouldn't be two sides. Both should be catered to... there shouldn't be two separate demographics."

"The world should aspire to not see technology in front of them," he said, picking up his white Nokia Lumia 920 in demonstration. "People still have their thumbs on phones. We should be able to make things happen without needing people to think about technology. Things should just work ... If we could eliminate that barrier, lift that technology veil, it would change lives... technology would be natural, you would be able to do things without even thinking."

In all our interviews, Maizels kept returning to Isaac Asimov as an inspiration. He wants machines to be smarter, to evolve to the point where we no longer realize they're complex creations, to interact with them in the most natural way possible. In some sense, we're almost there, with touchscreens and hand gestures, but Maizels wants to take it a step further.

He wants his sensors to be so universal that it creates a "new way of living." Perhaps it will.

[Image credit: Ronen Goldman]