For 24-year-old Quinten Hoskins, new federal guidelines suggesting that people wear homemade face coverings in public to fight the coronavirus outbreak seemed like a joke, and not a very funny one at that. “Can y’all imagine me walking in here with a bandana on my face?,” he asked a group of colleagues at the Milwaukee hospital where he works as a unit secretary. “It got extremely quiet,” he says. “Everybody understood where I was coming from.”

Those guidelines, shared by the Centers for Disease Control and Prevention (CDC), now call for Americans to wear cloth masks when in public, fashioned, for instance, from an old t-shirt or bandana. The suggestions are meant to reduce the spread of the virus while dissuading people from buying the medical-grade masks so badly needed in hospitals. But some black men, like Hoskins, are worried about the potential danger in wearing a homemade face covering.

“The image of a black man, especially a tall, dark-skinned black man with a bandana on his face, I just feel like the racial biases that will pop into people’s heads will be something that could very well be the end of my life,” says Hoskins, who’s continuing to wear a mask in public despite his concerns.

A handful of recent encounters between black men and law enforcement around the country have underscored the fraught racial politics of mask-wearing. In one video from Miami, an officer is seen handcuffing a black doctor who was wearing a protective mask outside his own home; the police chief is investigating the episode. Another video shows a police officer following two black men wearing surgical masks out of an Illinois Walmart; the police department later said the officer had asked for the men’s IDs after wrongly telling them that a city ordinance prohibited wearing masks.

These highly publicized incidents are now factoring into many Americans’ decisions over whether to wear a mask in public. Derrick Richardson, a 50-year-old San Francisco-area medical industry consultant, is particularly worried about his two adolescent sons. His concerns are amplified by his own life experience — in 2007, Richardson, then the director of a biotech startup, was walking home when he was stopped by a police officer, he says. She drew her gun on him, saying that he matched the description of someone robbing houses in the area. “It was a horrible experience. It was gut-wrenching,” he says.

“It doesn’t matter how many degrees I have, how much money I have,” says Richardson. “I’m still a black man in America, and the thought of walking down the street with a mask on scares me. I’m afraid of how people will perceive me and what might happen to me or one of my kids.” Richardson, who has since become involved with the Union City Human Relations Commission, says he believes the local police department has made notable progress in community relations in recent years. The local police chief says that, while the department still makes stops when they believe a crime may be in progress, they now try to make sure that when innocent people are stopped, they receive more information so they don’t feel they were unfairly targeted.

Keep up to date with our daily coronavirus newsletter by clicking here.

Several black men who spoke with TIME for this story say they are changing their behavior when wearing a face covering in public. Andrew Allen, a 44-year-old service advisor at a truck leasing company in Hillside, NJ, says he told his relatives to wear a mask for health reasons, but also recommended they only visit establishments where they are likely to be recognized. “I just tell them to stay home as much as possible, go to a local store that you’ve been to before, where there’s probably a familiar face,” says Allen. “Take your hand sanitizer, wear your mask, do your business and get out. That’s pretty much what we can do until things simmer down.” Hoskins said he tries not to stare at others too much. Richardson, who says he’s spent much of his life trying to appear non-threatening in predominantly white spaces, says he’s going to be “very conscious about the way I make my way around to be very mindful of my body language.”

Marc Morial, President of the National Urban League and former mayor of New Orleans, says concerns over mask wearing are “very real.” That people are being encouraged to make their own masks in the face of medical equipment shortages compounds the problem, he adds. “It would be different if there were a standardization around the masks,” Morial says. “But for people to use a bandana, it invites racial profiling. It conjures up every racial stereotype that we have to fight against when it comes to African American men and Latino men.”

Spotlight Story

Finding Hope: The TIME 100 Community on Navigating Our New Reality

Here’s advice from Margaret Atwood, Sundar Pichai, Stephen Curry, Tsai Ing-wen and others on how to get through the coronavirus pandemic

Anxiety about mask-wearing is yet another factor complicating COVID-19’s already devastating impact in America’s black communities. Early data appears to show that the virus is killing a disproportionate number of black people — in Milwaukee County, for instance, black victims represent more than 60% of deaths, though they make up only 27% of the overall population. The lopsided death toll is likely due to longstanding healthcare inequalities, higher population density and other factors that have made black and other minority communities more vulnerable to the outbreak. “The pattern that we’re observing with COVID-19 is not surprising,” says Courtney Cogburn, a professor of social work at Columbia University. “The degree of the difference is pretty shocking.”

While many American cities remain under varying states of lockdown, the time will eventually come for a gradual return to normal. Some people are worried that, if mask-wearing remains part of everyday life, it could result in a spike in racially-motivated incidents as more people are out together in public again. Richardson, for one, is preparing to talk with his family about staying safe. “We’re going to at some point have to have a pandemic black male survival sit-down,” he says.

John DeCarlo, a professor of criminal justice at the University of New Haven and a former police chief in Branford, CT, says it’s far from certain that mask wearing will result in a significant increase in racial incidents. But he believes it’s important to start having conversations on the topic now. “Things will be different if we are wearing masks,” he says. “We’re going to lose a level of communication that, whether we know it or not, we’ve depended on to read people’s intent.”

Police departments, DeCarlo says, should brief officers on issues around masks to avoid problems. But others are less certain such efforts would do much good, given people’s conscious and subconscious biases. “We could message and say, ‘Hey, everyone’s going to have on a mask, don’t treat them as a threat,’” says Cogburn. “But I don’t know if that will undo all the things that actually contribute to your decision-making about seeing a person as a threat or not.”

“The issue of the mask is a representation, it’s a symbol of a much bigger and complicated problem,” she adds.

For Richardson, facing uncertain months ahead, there seems to be little choice but to prepare for the worst. “I’m almost bracing myself and expecting that I’m going to have some negative experience before this is all over that is rooted in my race,” he says. “I’ve been at this for half a century — I think I know what’s coming.”

Please send any tips, leads, and stories to virus@time.com.

Write to Alejandro de la Garza at alejandro.delagarza@time.com.