The smooth side does indeed face the owner. But it's about maintenance, not appearance. If the fence faced the other way, you'd have to be standing in your neighbor's yard to fix your own fence if it has a broken board.
Of course, that doesn't mean that some people don't install their fences backward.