Anti-marijuana advocates needn’t worry. Just because weed is all out legal in 4 states, medically legal in more than half of the nation, debunked as a seriously harmful drug, and used by 49 percent of Americans, doesn’t mean there isn’t a good way to seriously inconvenience stoners’ lives.

Because even if lawmakers, doctors, and a majority of everybody else in the country is pretty chill with weed being legal, employers aren’t. That’s according to a new survey, conducted by the Society of Human Resource Management and reported by the Denver Post, which polled 632 HR managers in weed-legal states just how cool they were with their employees toking up.

More than 50 percent said they either currently have or plan to soon have policies prohibiting hiring cannabis users. A whopping 38 percent said they would refuse to hire a user, even if their weed usage was strictly medicinal in nature. Only 6 percent made a distinction between medical and recreational and said they would only reject users who use pot recreationally.

“There is what I consider to be a significant number of employers that are saying they wouldn’t hire an employee that uses marijuana,” Evren Esen, director of survey programs at SHRM, told the Post.

An employer’s decision to fire an employee based on their use of medical marijuana was legitimized by the Colorado Supreme Court earlier this year in the case of Brandon Coats. Coats was a customer service representative for DISH Network, who was let go from his position after a drug test revealed that Coats had been using marijuana, though only off duty and with a doctor’s recommendation to treat his leg spasms.

And if an employer can say that an MMJ patient is unfit to answer phones because they use marijuana in their off hours, then just about any employer could feasibly make the same case. By common sense standards, the court’s 6-0 unanimous vote to uphold DISH’s right to fire Coats seems unreasonable. But by legal standards, it’s logical. Since marijuana is still a federally controlled substance, it’s use is subject to a whole lot scrutiny that other medicines aren’t.

But all this might be slightly overstating the case of cannabis patient discrimination. The survey also found that fewer employers are currently drug testing their employees than were five years ago, with the number dropping from 55 percent in 2011 to less than 50 in 2015.

And only 21 percent of those polled said their company has had chronic problems with stoned employees, meaning they had more than 3 employees violate their marijuana policy in the last year.

“It doesn’t appear to be a really major problem,” Esen said. “It doesn’t seem like employees are going out there and rampantly using marijuana in greater numbers than before.”