# Pedestrians are a hard problem for self-driving cars - The never ending story



## jocker12 (May 11, 2017)

Pedestrians pose one of the biggest challenges for self-driving cars. Other cars mostly behave in predictable ways, but pedestrians can be more erratic. They can move in any direction, at any time, with little warning.

Fortunately, we humans are experts at understanding the behavior of other people. Human drivers can easily distinguish someone who is trying to cross the road in front of us from someone who is waiting for the bus or focused on his cell phone.

And that allows human beings to be smooth, confident drivers. If we pass a sidewalk full of pedestrians, we can tell at a glance if any of them are preparing to step in the road. If not, then we know it's safe to maintain our speed.

But encoding that kind of intuition into software isn't easy, which can lead to self-driving cars that are timid or erratic. A car that doesn't understand pedestrians will need to slow down every time it passes near one-since it can't rule out the possibility that the pedestrian might wander in front of the vehicle.

A new startup called Perceptive Automata is aiming to change that by using machine-learning techniques to give self-driving cars a rich understanding of pedestrians, cyclists, and other human beings around the car.

*Human intuition in software*
"We're building a module that allows autonomous vehicles to understand the state of mind of humans out on the road," says co-founder Sam Anthony. He told us that the software will "give autonomous vehicles the ability to look at a person and say, in a human-like way, 'This person wants to cross the road, this person knows that my car is there.'"

Ordinarily, machine-learning techniques train algorithms using data that can be measured objectively. But that's not really practical in a situation like this-it's not like the company can read the minds of pedestrians on the side of the road for its training data. Instead, Perceptive Automata relies on the subjective judgment of other human beings to provide the data used to train its algorithms.

The company asks human beings to watch video clips and then label the pedestrians in them, giving their best judgment about whether each pedestrian was trying to cross the street and whether he noticed the car. Perceptive Automata's engineers then use this dataset of labeled videos to train machine-learning algorithms to make the same kind of judgments.

The result, the company says, will be a software module that any self-driving carmaker can buy and drop into its existing autonomy stacks. Perceptive Automata argues self-driving carmakers should think of it as an additional sensor that can effectively read the minds of pedestrians around the vehicle. This "sensor" data can then be combined with data from cameras, lidars, and other hardware sensors to enable smoother, more confident driving.

An obvious question here is whether companies developing self-driving cars will really outsource this function to a third party like Perceptive Automata rather than developing the capability in-house. But Anthony told us that companies building self-driving cars have a lot on their plate and predicted that they'd welcome the ability to outsource this function.

"For someone who's doing the full stack, they're fighting so many fires just to get their testing fleet out there," Anthony said. There's an art to efficiently collecting data from human beings about other human beings, he said, and it's pretty far afield from the other tasks self-driving car makers are tackling.

*Is it really necessary to model pedestrian thinking?*
But self-driving car companies like Waymo and Uber are collecting millions of miles of sensor data from test driving. I asked Anthony whether they could use this sensor data to directly predict pedestrian behavior-simply by seeing which pedestrian behaviors tend to precede actually crossing the street.

But Anthony argues that it's much more difficult to predict whether a pedestrian _will_cross the street if you don't have a model of whether the pedestrian _wants_ to cross the street. After all, a pedestrian might stand at the side of the road for several seconds waiting for the right moment to cross. What a self-driving car ultimately cares about is whether the pedestrian is trying to cross the street, not just whether he or she is going to do so in the very next second.

I didn't find this entirely persuasive. It seems like with enough data-and Waymo now has 8 million miles of data-statistically predicting which pedestrian attributes and behaviors are correlated with street-crossing behavior should be possible. Data about the pedestrians' state of mind would certainly be a "nice to have" feature, but I'm not convinced that it's a must-have feature.

Other researchers are working on methods to predict pedestrian movements based entirely on their observed behavior-without asking humans to annotate training data with pedestrians' state of mind. It's possible this kind of technique will work well enough to allow pedestrian behavior to be inferred based solely on the sensor data companies already have.

Still, Anthony says that Perceptive Automata has attracted significant interest. Nvidia has described Perceptive Automata as a partner. Current customers include carmakers and their tier 1 suppliers, as well as some startups, Anthony told Ars. The company hasn't yet publicly identified any of its customers, however.

https://arstechnica.com/cars/2018/0...-cars-intuitive-understanding-of-pedestrians/


----------



## heynow321 (Sep 3, 2015)

Yeah good luck with that


----------



## uberdriverfornow (Jan 10, 2016)

Or you can just use a car with a human driver and not have to worry about that.


----------



## 404NofFound (Jun 13, 2018)

And to read the mind of the insane person who is about to push you into oncoming traffic!


----------



## jocker12 (May 11, 2017)

To me is funny how they are still at square one, still trying to figure out how to detect and avoid pedestrians.

The main problem they have with the software (improperly called AI) is that is NOT learning from its day by day operations and interactions or previous knowledge. It only learns from datasets, and that is exactly what the article says - "The company asks human beings to watch video clips and then label the pedestrians in them, giving their best judgment about whether each pedestrian was trying to cross the street and whether he noticed the car. Perceptive Automata's engineers then use *this dataset of labeled videos to train machine-learning algorithms* to make the same kind of judgments."

And understanding humans is only one problem they have, one top of understanding deer thinking, cows and sheep thinking, or even dogs or cats thinking.


----------



## jocker12 (May 11, 2017)

This is what they are dealing with -






The human driver gets through construction and by unpredictable pedestrians, while a robot will probably stop not knowing where those pedestrians next move will take them, and what those white and orange construction obstacles are all about.


----------



## 404NofFound (Jun 13, 2018)

jocker12 said:


> This is what they are dealing with -
> 
> 
> 
> ...


The construction guys also have those two sided signs to control traffic.


----------



## jocker12 (May 11, 2017)

404NofFound said:


> The construction guys also have those two sided signs to control traffic.


Yup, and all this unpredictable movement is confusing for the software, that is not learning from interaction with the environment but from data sets. As long as all the elements adjust their reactions to surrounding random events, the robots are useless.

If every single thing around it'll automated, than a computer will have much less trouble moving around.


----------



## the_king_of_$3.18 (Jul 28, 2018)

This is why the usefulness of SDCs will be limited, for the time being, to specific markets only. Phoenix is such a market, having very few pedestrians. A Mars colony would be another such market.


----------



## jocker12 (May 11, 2017)

the_king_of_$3.18 said:


> This is why the usefulness of SDCs will be limited, for the time being, to specific markets only. Phoenix is such a market, having very few pedestrians. A Mars colony would be another such market.


The Mars Rover is already there, and doing great moving at a maximum speed of 0.09 mph.


----------



## the_king_of_$3.18 (Jul 28, 2018)

Until the technology is fully capable of living up to its promise, municipal governments will for safety reasons have to choose between banning SDCs and banning pedestrians. For most places the choice is obvious, but in an SDC-friendly place like Arizona there will surely be a few towns or cities which will really really want to have SDCs before everywhere else - and will adjust civic priorities accordingly.


----------



## Stevie The magic Unicorn (Apr 3, 2018)

Not entirely related but...

These things will be a breeze to screw with


----------



## jocker12 (May 11, 2017)

Self driving robots will be trashed for the same reasons electric scooters are being trashed in SF - https://www.sfgate.com/bayarea/arti...rs-lime-spin-bird-12849190.php#photo-15437172

Any irritated PAX or pedestrian could cover their faces and pop the tires or smash the windshield or any window of a SDC with a sharp heavy object.


----------



## everythingsuber (Sep 29, 2015)

jocker12 said:


> Self driving robots will be trashed for the same reasons electric scooters are being trashed in SF - https://www.sfgate.com/bayarea/arti...rs-lime-spin-bird-12849190.php#photo-15437172
> 
> Any irritated PAX or pedestrian could cover their faces and pop the tires or smash the windshield or any window of a SDC with a sharp heavy object.


I would still be more concerned about what people inside the cars do to them than those outside. It will end up like a public toilet on wheels.


----------



## jocker12 (May 11, 2017)

everythingsuber said:


> I would still be more concerned about what people inside the cars do to them than those outside. It will end up like a public toilet on wheels.


I agree, but that is not a concern. It's the end of Pool.


----------

