Whilst it is considered respectful to stare at anything else but the person you are talking to in some cultures, this may just be why you never got called back after that interview