You haven't said which device you're using, so I'll assume iPhone.
In the Audible app (on my iPhone), when playing a book, right above the play button, it says "connect to device", which then lets me chose "AirPlay & Bluetooth Devices" and I can stream audio directly to my Apple TV.