A ‘smart’ walking stick can help the visually impaired shop for groceries and find a seat | CU Boulder Today

Engineers at CU Boulder are taking advantage of advances in artificial intelligence to develop a new type of walking stick for people who are blind or have low vision.

Think of it as assistive technology meets Silicon Valley.

Researchers say their “smart” walking stick could one day help blind people navigate tasks in a world designed for sighted people — from shopping for a box of cereal at the grocery store to choosing a private place to sit in a crowded cafeteria.

“I really enjoy grocery shopping and spend a lot of time in the store,” said Shivendra Agrawal, a doctoral student in the Department of Computer Science. “A lot of people can’t do that, however, and it can be really limiting. We think that’s a solvable problem.”

In a study published in October, Agrawal and colleagues at the Collaborative Artificial Intelligence and Robotics Laboratory came close to solving it.

The team’s walking stick is similar to the white and red sticks you can buy at Walmart. But it also includes some extra functionality: Using a camera and computer vision technology, the walking stick maps and indexes the world around it. It then directs users by using vibrations in the handle and spoken directions, such as “Reach slightly to your right.”

Agrawal said the device is not meant to be a replacement for designing places like grocery stores to be more accessible. But he hopes his team’s prototype will show that, in some cases, AI can help millions of Americans become more independent.

“Artificial intelligence and computer vision are improving, and people are using them to build self-driving cars and similar inventions,” Agrawal said. “But these technologies also have the potential to improve the quality of life for many people.”

sat down

Agrawal and his colleagues first explored this possibility by tackling a familiar problem: Where do I sit?

He said, “Imagine you are in a coffee shop.” “You don’t want to just sit anywhere. You usually take a seat near the walls to maintain your privacy, and you usually don’t like sitting face-to-face with a stranger.”

Previous research has indicated that making these kinds of decisions is a priority for people who are blind or have low vision. To see if their smart walking stick could help, the researchers set up a café of sorts in their lab—complete with several chairs, customers, and a few obstacles.

Study subjects strapped to a backpack with a laptop inside and picked up a smart walking stick. They turned to scan the room with a camera attached near the handle of the wand. Like a self-driving car, algorithms running inside the laptop computer identified the various features in the room and then calculated the route to an ideal seat.

Picture of a shelf containing several different boxes of cereal.  Each square is marked with a red or green square and is scored with numbers ranging from 0.36 to 0.91.

A computer vision algorithm scores dozens of boxes of cereal to identify a target product—in this case, a box of Kashi GO Coconut Almond Crunch. (Credit: AI and Robotics Collaborative Lab)

The team announced its findings this fall at the International Conference on Robotics and Intelligent Systems in Kyoto, Japan. Study researchers included Bradley Hayes, assistant professor of computer science, and doctoral student Mary Etta West.

The study showed promising results: the subjects were able to find the right chair in 10 out of 12 trials of varying levels of difficulty. Until now, all sighted people wore blindfolds. But the researchers plan to evaluate and improve their device with working people who are blind or have low vision once the technology becomes more reliable.

“Shivendra’s work is the perfect blend of technical innovation and impactful application, going beyond navigation to bring about advances in unexplored areas, such as helping people with visual impairments adhere to social norms or finding and picking up objects,” Hayes said.

Let’s go shopping

Next up for the group: grocery shopping.

In new research, which the team has not yet published, Agrawal and his colleagues have adapted their device for a task that would be daunting for anyone: finding and picking up products in aisles filled with dozens of similar-feeling options.

Once again, the team created a makeshift environment in their lab: this time, a grocery rack filled with many different types of grains. The researchers created a database of images of products, such as boxes of Honey Nut Cheerios or Apple Jacks, in their programs. The study participants then used a walking stick to scan the shelf, searching for the product they wanted.

“It assigns a score to the things that are there, and chooses the most likely product,” said Agrawal. The system then issues commands such as “move slightly to your left”.

He added that it will be some time before the team’s walking stick gets into the hands of real shoppers. The group, for example, wants to make the system more compact, and design it so that it can be operated from a standard smartphone attached to a stick.

But researchers in human-robot interaction also hope their initial findings will inspire other engineers to rethink the power of robotics and artificial intelligence.

“Our goal is to make this technology mature but also to attract other researchers in this field of assistive robotics,” said Agrawal. “We believe that assistive robots have the potential to change the world.”

Leave a Comment