Stroll into an HSBC Bank branch office in New York, Seattle, Beverly Hills, or Toronto (see below), and you’re likely to encounter a 4-foot-tall, all-white, 60-pound, semi-humanoid robot that’s clearly trying to get your attention by using the same verbal and nonverbal cues that we’re all familiar with. As you approach Pepper, developed by SoftBank Robotics, it might make a humming noise to indicate that it has verbal abilities and wants to start a conversation. It sways back and forth to display a friendly, welcoming sort of animation. It might do a little dance. And it nods its head and moves its arms to approximate human body language.

Once you get closer, Pepper’s facial recognition and social skills kick in. Pepper looks right at you. A pale greenish pulsating glow accentuates Pepper’s round black eyes that sit recessed in its head. Pepper speaks: “Welcome to HSBC. How can I help you today?” And that’s where the fun starts.

Studying Real-world Interactions

Matt Willis, Ph.D., design and HRI lead at SoftBank, has been intently studying the real-world interactions between Pepper and bank customers in order to continue to improve the effectiveness of the robot. He says initial reactions to Pepper run the gamut; some people stop and watch from afar, some run right up, some just wave, some start talking, some try to shake hands, and many whip out their phones to take selfies and videos.

You can always tell who has Alexa in their house, because those customers ask the robot for its name and then use “Pepper” as the trigger word, even though that’s not necessary. Willis says that when it comes to human-robot interactions, there’s a learning curve on both sides.

The design challenges that Willis faces are far less about the technological aspects of building a robot than about the psychological, behavioral, linguistic and social dimensions of the human-robot experience. And Pepper has a very specific use case: The goal is not to just be entertaining; the goal is to provide real business value.

At HSBC, Pepper’s role is to dispense information about banking products and services, and to answer lower-level customer queries. When the situation calls for it, Pepper can refer customers to the appropriate bank employee. In other words, Pepper serves as a sort of walking, talking, interactive information kiosk.

Non-threatening, Yet Intelligent

One of the key design criteria was making sure that Pepper was not imposing, threatening or off-putting in its physical appearance, particularly for a public setting like a bank where visitors can run the gamut from children to the elderly. That’s why Pepper is short, has a soft, rounded shape and a childlike appearance.

In addition, there was no attempt to make Pepper look human. “Pepper is a robot and doesn’t pretend to be more than a robot,” Willis says. “Pepper is not trying to be human – it’s trying to be social.” This approach avoids the whole ‘uncanny valley’ issue that other humanoid robots that look “too human” could face.

In addition, the team worked on Pepper’s voice to make it different from the halting, monotone, and usually female voice that humans hear when receiving a robocall or encounter a call center bot.  Pepper is different. A team in Paris developed a unique voice that matches Pepper’s physical appearance; in other words, Pepper has a clearly non-human voice that is meant to be friendly and to show emotions, particularly enthusiasm and excitement. At the same time, the voice isn’t too cartoony, because it has to convey to the customer that Pepper has intelligence, and is capable of discussing serious topics like banking products and services.

Gesture and movement are additional challenges, says Willis. Pepper needs to find the right balance between attracting customers and not being pushy. So, once Pepper grabs someone’s attention, it doesn’t invade anyone’s personal space; it’s up the customer to approach the robot.

In addition, the robot’s gestures and hand movements need to take into account all of the potential misunderstandings that occur in conversations. For example, if a customer asks a question and Pepper nods, does it mean that Pepper understood the question, or does it mean that Pepper is answering ‘Yes’? Willis said these are the subtleties that his team is constantly refining and improving.

Natural Language Barriers

If you think about the times when human conversation results in some sort of confusion or miscommunication, you can understand the challenge with robot communication. The last thing the SoftBank team wanted was a customer getting frustrated or exasperated with Pepper.

To address this, Pepper was designed to be bi-modal. It has a touch-screen tablet attached to its chest that displays several options for customers, who have the choice of interacting with Pepper verbally, via the tablet or switching seamlessly back and forth. For example, Pepper might point to the tablet and say, “You can select one of these options or just ask me a question.”

Willis says he tapped into the expertise of linguists to help Pepper structure sentences in a conversational manner, and to vary its responses so it doesn’t sound too repetitive. Other challenges include what to do when Pepper doesn’t understand a question, or understands the question but doesn’t have an answer. Pepper might say something like, “I don’t have an answer to that, but here are some other bank services that you might be interested in.” Or Pepper might call for back-up in the form of a human bank employee.

Results and the Future

While people are concerned that robots might take away their jobs, Pepper is generating so much foot traffic in HSBC branches that the bank is actually boosting headcount. HSBC reports that Pepper is driving an increase in ATM transaction volumes and a rise in new credit card applications. One other important caveat: Pepper does not collect any customer data or personal banking information; it simply provides generic information on bank products and services.

Willis says the possibilities are endless as people become more accustomed to humanoid robots, and as Pepper gets better at understanding context, tone of voice, facial expressions, and body language.

For example, in a retail setting, Pepper might answer questions on where to find a certain item. But an even more sophisticated solution might link Pepper to real-time inventory levels, so Pepper can also tell a customer that a particular item is out of stock before the customer tries to find the item on their own. And in the banking scenario, Pepper’s facial recognition technology could be linked to a customer database, so that Pepper can welcome repeat customers by name and offer them personalized services.

Willis sums up his mission this way: “The challenge is understanding how we can develop robots that bring value to the world.”