According to the course, golfers are liable and most of the time the golfer's own home owner's insurance will cover the damage to property. So yeah, you're a world class a-hole if you damage someone's property, have insurance that would make that person whole at no cost to yourself, police the ball by entering a fenced pool area, and then don't take responsibility for the damage.
Furthermore, you suck at golf to hit my house period. We're not talking an errant shot. We're talking a line drive way out of bounds.