With more than 15,000 citations, 125 technical papers and more than 50 patents related to robotics and computer vision technologies, James Kuffner, Ph.D., is one of the most highly cited authors in the field. But he is also a proven entrepreneur and inventor, having helped create the rapidly exploring random tree (RRT) algorithm and coining the term “cloud robotics” to describe how network-connected robots could take advantage of distributed computation and data stored in the cloud.

Joanne Pransky, associate editor for Industrial Robot Journal, recently spoke with Kuffner about his career in the robotics industry, Silicon Valley, and the early days of motion planning for robots.

The full interview is available free to Robotics Business Review readers until March 31. Here is an excerpt:

Early opportunities

Pransky: Did success always come naturally to you? Did you have to work at it and what professional or personal aspects have you had to struggle with?

Kuffner: I was just an ordinary kid who grew up in Portland, Oregon. My first job, when I was 12 years old, was picking strawberries in the summer. I always loved robots and science fiction. I read books a great deal and I tinkered with home computers. My first computer was a Radio Shack TRS-80 with 16 KB of RAM and I stored computer programs on a tape drive that I wrote mostly in Assembly and Basic. But I really taught myself, along with some of my friends, how to write code mostly to create games. Though I did not actually spend a lot of time playing the games I wrote, I certainly learned a lot about software engineering and programming.

I was fortunate enough to get accepted to Stanford and I spent 10 years there. It was a great environment. I lived through the Internet Revolution. In 1995, when I was a second-year grad student, many new companies were created, and I joined the Ph.D. program at Stanford along with 29 others. I graduated in 1999 with only 14 others in my class. It was a historic low for Stanford, but the joke was that the ones who did not graduate were the smart ones because they started companies like Google and Yahoo.

We were the ones that stuck it out, but I also got to start and co-found companies while I was in grad school. I learned so much in Silicon Valley at a time when the Internet was taking off. There was suddenly this huge opportunity that your potential to create something was only limited by your imagination, the tools you had at your disposal and the resources that you were given. Software is a very creative process.

The logo for our company, TRI-AD (Toyota Research Institute-Advanced Development, Inc.) is based on origami. It is craftsmanship of paper, but it is art and engineering together. What I love about it is that from a piece of paper with origami, you can create anything. And it is the same way with software from a blank text buffer. I can input code and create anything, as there is unlimited potential now when you have code that is not just stuck in the embedded device that you are programming. Now you have connectivity, and you can spread software and data instantly between all of the connected devices in the world.

That really led to my thinking a lot about cloud robotics. I still think that fleet learning is the future. The idea that you could program a robot and if it became proficient at that task, you could then instantly transmit that skill to any other robot so that over time all of the robots become more skillful and more robust. It is really about taking and generalizing experience. Instead of building one robot and running it for 10,000 hours to collect data, I can have 100 robots run for 100 hours and collect the same amount of data, so much more accelerated, and they can be distributed. To me, that is the untapped potential of how we can make more skillful machines that can do tasks more reliably and more robustly, through data sharing and sharing experience.

Pransky: What led you to co-found the companies that you started when you were younger and how did they end up doing?

Kuffner: The first company, Motion Factory, actually came out of some research I was doing with some Stanford colleagues. I had worked on an animation that was generating and synthesizing motion for human characters. We found that just around that time, computer graphics were taking off and people were building special purpose chips that were able to produce 3D imagery. Silicon Graphics was the notable company that pioneered real-time graphics. That meant people were able to process more and more geometry and depict artificial scenes in real time with real-time motion. But they wanted to depict, for example, human characters, which were all hand-animated. There were offline animations like Pixar created in their new films such as Toy Story with 3D human animation. But at that time, in 1994, it took an hour to render one frame in the original Toy Story.

We were seeing these graphics cards that were being built that were doing 30 frames a second of thousands of triangles and more and more complicated scenes. At the same time, motion capture technology was just starting, and people were using optical motion capture or magnetic motion capture to capture human performances and embed them as clips into games. We thought people are going to need to generate realistic human motion and that we could build algorithms that will help people to do this. And that became our company.

We were basically a middleware company, mostly for real-time graphics like games. One of our famous releases was Prince of Persia 3D. That game used our software, but I decided to go back to school and finish my Ph.D. Eventually Motion Factory was acquired by Avid Technology in 2000, which was part of Softimage, a Microsoft company. It was really a great experience and I learned a lot about starting companies in Silicon Valley.

I worked on a couple of others. Robot Autonomy was a startup that I built to mostly help companies think about how they can use motion planning and automate autonomy in everything from factories to humanoid robots.

I was involved with another company, called Jibbigo, that I worked on with my colleague at Carnegie Mellon, Alex Waibel, and his students. It was the first iPhone app that could do real-time speech-to-speech translation. In 2008, we basically built an app that could record English, and in real-time, output spoken Japanese or spoken Spanish. We had a couple of other languages, but that company was eventually acquired by Facebook.

Industry vs. academia

Pransky: How do you like working in the industry compared to academia?

Kuffner: It has often been said that, on the one hand, there is the power of the paper and the power of the product. I loved my job as a professor, especially every year, when young, bright-eyed, bushy-tailed kids would come to the campus and their energy level was just infectious. They were so excited to learn and to try new things and to really absorb the environment of learning. That to me is the wonderful part about our universities.

But after a decade of mostly writing papers and hoping that somebody would read those papers and turn it into a product to actually impact the world, I have now been really enjoying skipping that step and just working on the product. I think both are important. The reality is that there is no human advance in the history of our planet that has not been a co-evolution of ideas and practical realization of those ideas. It is the research and development that go hand-in-hand. You need both, and that is why when we created Toyota Research Institute four years ago and I was the CTO, we partnered with universities so that we could try and assemble some of the best minds to think about problems that we felt were important.

Pransky: What do you think Ph.D. and masters engineering students should do in school to best prepare themselves for the real world of engineering?

Kuffner: In computer science, I feel that there are many systems engineering skills. If you are interested in robotics in particular, it is a unique blend of mechanical systems, electrical systems and algorithms and computation and also human-machine interaction. In many ways, I believe you can teach all of computer science by having students build robots. That is why I love the STEM activities for younger kids. I love how robots are now entering the curriculum in middle school and high schools and universities and I always like to say, if a student wants to make a difference, engineering is the right way.

Pransky: What has been the proudest moment in your long and illustrious career?

Kuffner: It has been wonderful to raise kids. There was a joke going around among all of the roboticists in the past 25 years that if you want to create an intelligent being, the easiest way is just to have a baby. It is actually very humbling for a scientist because we are trying to create intelligence. We’re trying to create machines that can achieve human performance or superhuman performance on different tasks.

I seriously believe that if technology does not advance, then the future for humanity might look a bit grim. But technology is the wild card. It is the way that human society can surpass limitations and augment our ability. The optimism of the future is really around augmenting human ability, and we must never forget that technology is to help people. I believe we should always not create technology just for technology’s sake. We should think about how it helps people. That gives me great hope that because technology is improving, our planet has a bright future.

Pransky: If you had a magic lamp, what technical solution would you wish for?

Kuffner: I feel that reliable perception is still on the critical path to a lot of useful things. I used to work in motion planning, and it turned out that even if I had the perfect motion planning algorithm, if the perception system was giving the wrong information about the environment, the robot would do something incorrect. Thus a good motion planning system depends on having a good perception system. Now that we have these new sensing modalities, we can use them to process in real time. The robots in real time are going to be able to process their environment at scale, and then suddenly it will unlock all of these tasks that are unstructured. We have seen great advances in deep learning and we have seen dramatic improvements in vision and lidar, and I think perception is just going to keep getting better.

Read the complete interview here.

More Essential Interviews:

• Brian Carlisle, Precise Automation
• Hod Lipson, Columbia University
• Howard Chizeck, Olis Robotics
• Maja Mataric, USC and Embodied
• Ken Goldberg, UC Berkeley
• Aaron Edsinger, Hello Robot Inc.
• Robin Murphy, Roboticists Without Borders
• Vecna Robotics’ Daniel Theobald Talks About Doing Well by Doing Good
• CMU’s Howie Choset on Robotics Collaboration
• William ‘Red’ Whittaker, Field Robotics Pioneer and Entrepreneur