Definitely go for it. As long as they don't look like slippers (i.e. not fluffy or tartan) nobody is going to notice - or care, I expect, if they did. It is only sense not to wear the same shoes indoors as out of doors and your feet need a break. I wore black ballet flats at the office for years!