We’ve all heard the phrase, or maybe we’ve even used it to describe someone: “They have an honest face.” Something about the person just seems genuine, trustworthy. Unfortunately, it’s not a kind of honest vibe or trustworthy je ne sais quoi we’re sensing — it’s simply familiarity. We tend to trust strangers who physically resemble previously known, honest individuals, and mistrust those who resemble people we know to be untrustworthy.

“Our study reveals that strangers are distrusted even when they only minimally resemble someone previously associated with immoral behavior,” said Oriel FeldmanHall, who led research as a post-doctoral fellow at NYU. “Like Pavlov’s dog, who, despite being conditioned on a single bell, continues to salivate to bells that have similar tones, we use information about a person’s moral character, in this case whether they can be trusted, as a basic Pavlovian learning mechanism in order to make judgments about strangers.”

The researchers conducted a series of experiments centering on a trust game in which participants make a series of decisions about their partners’ trustworthiness — in this case, deciding whether to entrust their money with three different players who were represented by facial images.

Here, the subjects knew that any money they invested would be multiplied four times and that the other player could then either share the money back with the subject (reciprocate) or keep the money for himself (defect). Each player was highly trustworthy (reciprocated 93 percent of the time), somewhat trustworthy (reciprocated 60 percent of the time), or not at all trustworthy (reciprocated 7 percent of the time).

In a second task, the same subjects were asked to select new partners for another game. However, unbeknownst to the subjects, the face of each potential new partner was morphed, to varying degrees, with one of the three original players so the new partners bore some physical resemblance to the previous ones.

Even though the subjects were not consciously aware that the strangers (i.e., the new partners) resembled those they previously encountered, subjects consistently preferred to play with strangers who resembled the original player they previously learned was trustworthy and avoided playing with strangers resembling the earlier untrustworthy player. Moreover, these decisions to trust or distrust strangers uncovered an interesting and sophisticated gradient: trust steadily increased the more the stranger looked like the trustworthy partner from the previous experiment and steadily decreased the more the stranger looked like the untrustworthy one.

The researchers also found in the second task, the subjects’ brains tapped the same neurological regions involved in learning about the partner in the first task, including the amygdala — a region that plays a large role in emotional learning. The greater the similarity in neural activity between initially learning about an untrustworthy player and deciding to trust a stranger, the more subjects refused to trust the stranger.

“We make decisions about a stranger’s reputation without any direct or explicit information about them based on their similarity to others we’ve encountered, even when we’re unaware of this resemblance,” adds Elizabeth Phelps, an NYU psychologist and the paper’s senior author. “This shows our brains deploy a learning mechanism in which moral information encoded from past experiences guides future choices.”

If only it were a reliable learning mechanism, too. It’s one thing to have an honest face; it’s another to be trustworthy.