With labor shortages continuing to plague e-commerce fulfillment centers, and consumer demand for products increasing, companies are turning to robotics to help them pick orders more efficiently and quickly.

The latest advances include using machine learning and other AI techniques to make sure a robot can correctly and quickly pick an object it’s never seen before. Based in San Francisco, Osaro is an industrial-grade AI software company that specializes in vision and control systems for robots. The company’s OsaroPick offering integrates with automated storage and retrieval systems (ASRS) to perform various picking and placing tasks, enabling fully automated distribution centers.

Robotics Business Review spoke with Bastiane Huang, a product manager at Osaro, about the trends around AI-enabled robotic picking, challenges in warehouses beyond machine learning, and new industries where AI-enabled robotics will thrive.

AI breakthroughs

Q: It feels like piece-picking and AI software has really advanced over the last few years. What breakthroughs and technology innovations have occurred to allow for this advancement?

Huang: We’ve seen plenty of pilot projects with AI-enabled robots with conditional or high autonomy (Level 3/4). Warehouse piece-picking is a good example. In shipping warehouses, human workers need to pick and place millions of different products into boxes based on customer requirements. Traditional computer vision cannot handle such a wide variety of objects, because each item needs to be registered, and robots need to be programmed beforehand.

Machine learning (ML), deep learning and reinforcement learning now enable robots to learn to handle various objects with minimal help from humans. There might be some items that robots have never encountered before, and the machines will need assistance or demonstrations from human workers (Level 3 autonomy). But the algorithm will improve and get closer to full autonomy as the robots collect more data and learn from trial and error (Level 4).

The most important difference that machine learning brings to robotics is moving away from open-loop programming that relies on human data, to closed-loop learning from data. You don’t really see the difference between these two modes if the robot only does one thing. But if the robot needs to handle several tasks, or respond to humans or changes in the environment, it needs certain levels of autonomy.

In addition, cloud services and affordable hardware such as GPUs, cameras, and robotic arms have also accelerated the commercialization of AI-enabled robotics solutions.

Partnerships, integration

Q: Does Osaro provide the complete package for an application (hardware, grippers, software), or do they partner up with other companies to present the application for a customer? We realize that Osaro makes the software, but do they purchase the hardware themselves and then re-sell the system as a package, or is it a joint sale to a customer?

Huang: We don’t manufacture any hardware, but we make sure that our software is hardware agnostic, compatible with all major robotic platforms, grippers, sensors, and warehouse management systems (WMS). At Osaro, our mission is to build ML software that enables robots to learn and adapt to changes in the environment. Essentially we are building brains for robots, making robots smarter and more flexible.

We also partner with system integrators, working with many of the top 20 material handling equipment companies — firms that account for more than 85% of the global warehouse automation market. Some system integrators prefer purchasing the hardware themselves, and some don’t. We offer various options for customers. In addition to working with these system integrators, we also partner with top robotics companies.

Q: What challenges has the team faced in getting the software and application to pick objects correctly, swiftly, and accurately?

Huang: Deep learning (DL) is very good at learning representations using deep neural networks. DL technology is extremely powerful in supervised learning problems, allowing us to deliver great results without prior domain understanding for feature extraction.

However, this requires a large amount of labeled data, and acquiring data in the physical world can be expensive and time consuming. That’s why we are paying close attention to research areas that include meta learning, transfer learning, and imitation learning to enable better scalability and data efficiency for our machine learning models. We are also investing in more powerful simulation tools to accelerate the training process.

There are other challenges, which are not necessarily with machine learning, but definitely important. For example, real-world sensors are often noisy, failing to recognize objects with reflective surfaces or translucent packages; this is particularly a problem for traditional machine vision with 3D structured light sensors.  We address this issue by training deep architectures with labeled data collected using actual robot rollouts. We also use semantic segmentation, background recognition, and techniques like fully convolutional neural networks (FCNs) to learn rich feature representation at the pixel level, so we can recognize challenging objects with off-the-shelf commodity cameras.

A big deployment issue is Internet access at warehouses. It’s not always stable or available. Some customers prefer not to connect robots or machines to the Internet for security reasons. Besides bandwidth and connectivity constraints, the ML software needs to be integrated with surrounding systems, including robot arms, end effectors, sensors, and WMS. The robotics industry is extremely fragmented. Every robot maker has their proprietary interface and control languages, so it takes time and resources to complete the integration. On the other hand, customers want flexibility. Each system integrator has robots, grippers, and sensors that they are more familiar with, and would prefer to work with those.

All these issues add to the complexity of the problem we’re trying to jointly solve with integrators, and also forces us to simplify our solution to manage unexpected road blocks.

Picking challenges

Q: What types of objects are more difficult for a robot to pick? Does the company anticipate overcoming these difficulties, or will it just be a case where some things need to be picked by human workers? For those difficult objects, is it a question of software needing to improve, or does it need to be something where the end-of-arm tool / gripper needs to be adjusted?

Huang: Items with translucent packages, for example, are difficult to recognize by vision sensors. These issues can potentially be addressed with more training data.

What isn’t usually mentioned around AI solutions is that machine learning actually gets more expensive with higher accuracy. Furthermore, some items are simply not suction-able or pickable with current grippers. We believe a more pragmatic approach is to have machines do 80% to 90% of the work, and have people handle the rest, including edge cases.

Take warehouse picking as an example – with current technology, robots can handle most picking tasks, but they still need humans to help with error recovery and challenging tasks such as tightly packing items in boxes. This 80/20 strategy of automation to human intervention ensures companies can reduce costs on labor and system integration.  Picking algorithms will also improve over time as they learn from human training data.

Q: Which markets (e-commerce, advanced manufacturing, food assembly) has expressed the highest interest in Osaro’s solutions? Which applications (pick/place, sorter induction, kitting, packing, assembly) have also shown the greatest interest?

Huang: Our first product is piece-picking robotics software for e-commerce fulfillment centers because we see the most interest in this segment. Osaro builds end-to-end perception and control software for robots. We also work with customers on projects in manufacturing and food industries.

As a McKinsey report suggests, there’s an estimated $766 billion total wages in the U.S. for predictable physical work, which is more likely to be automated by robots. The top 3 markets where the most predictable physical work resides are 1) accommodation and food, 2) manufacturing, 3) transportation and warehousing. We are facing a global problem of shrinking labor forces driving up needs for automation. Especially for countries like Japan, its population is expected to shrink by 24% by 2050, and its working-age population is set to decline at an even faster pace than the overall population. Labor shortages drive up salaries in industries like e-commerce fulfillment, where the tasks are highly repetitive and openings are hard to fill. Almost all of the third-party logistics companies we talked to told us that they cannot find enough people for the Christmas season.

In a typical warehouse, an ASRS retrieves and delivers required SKUs directly to the order picker’s workstation in a designated SKU donor tote. Pickers pick the object into the box based on customers’ orders. This job is both stationary and highly repetitive. The turnover rate is high due to poor ergonomics, bodily strain and boredom, and U.S. companies lose as much as $11 billion every year to replace employees. That’s why at Osaro, we focus on automating this warehouse piece picking process with machine learning-enabled robotics arms.

Q: In the next few years, what advances do you expect to see from the company and its applications? For example, higher speed and accuracy for its existing applications, or do you see new applications and use cases being deployed?

Huang: Warehouse piece picking is obviously the first market to be disrupted by AI-enabled robots, not only because of the high demand, but also because it’s a relatively simplified problem compared to manufacturing, assembly or food. The tasks are not too complex, and it’s fault tolerant in a more structured environment.

Over the next few years, we expect that we will expand to other industries, including food, automotive, and electronics manufacturing, as the technology improves in its precision and adaptability.

Q: What does it mean for the company to be named to the RBR50?

Huang: Osaro is very proud to be included in the RBR50 report. This report highlights a lot of innovative companies, and we are honored to be considered among them.