I wear my shoes around the house sometimes when it's cold or if I just haven't gotten around to taking them off. I have them on right now, in fact. Otherwise, I go barefoot.
When I got to someone else's house, I leave my shoes on unless I know them really well, or I'm invited to take them off. I consider it rude to take them off otherwise. It presumes an informality that may not exist. I recognize that in other places removing your shoes might be a formal custom, but if you're in the south, keep your shoes on unless you have a casual relationship with the person.