Should being unfaithful be private business and not stray into your work life?

The top man at the CIA lost his job for "playing away" - but should being unfaithful be simply private business and not stray into your work life?