I think there's a meaningful difference between "some people have to come in, and the only way to be fair is to make everyone come in, whether or not that makes sense for their jobs" and spreading the burden of the X amount of work that really does benefit from being in person.

It sounds like what you're trying to avoid is the situation where Alice has to be in the office two days a week no matter what, so she gets stuck coming in every day to handle the in-person queries, while Bob's other work can be done remotely, so he doesn't have to do a share of the in-person stuff.

The only things I'm sure of are (a) if some people genuinely prefer working in the office, you don't have to send them home in the interests of abstract fairness, (b) you probably need a definition of "clinically vulnerable" that doesn't translate to "whoever complains loudest" or "whoever management likes." And that should probably be a university-wide guideline.