What about your relationship?
Infidelity isn't somthing to be taken lightly and the consequences can be severe. A 'bit on the side' doesn't just appear out of nowhere: it's a choice you make at a certain point in your life. You need to be aware of the fact that the partner you cheat on could decide to end your relationship even if you don't want that to happen.
What if you fall in love with your lover and end up having to choose between the two? Worse: your lover drops you and your partner dumps you when he finds out about the affair. All just for a bit of fun? Is it really worth it?
If you have children
Whether you have good 'reason' to have an affair or not, your children don't need to know the ins and outs. And if they find out, they'll not only be angry with you for cheating on their dad: they'll also be badly affected by the changes in the way they see their parents' relationship. If you don't want to make the effort to hide your indescretions from him, at least spare the children. They didn't ask for any of it to happen.