I sincerely doubt anyone will notice. I wore slippers in the office when we had awful weather and I'd walked into work in snow boots which I did not want to wear all day and literally *one* person noticed, and those were fake fur booties so arguably 'fluffy'. And that was one person out of an entire floor of open plan office, so maybe 30 people?