Well, there you go - the issue is with the company and the irreplacable boss. He is the weakest link obviously. Imagine he gets into a car crash, gets hit by a bus. Suddenly all his knowledge is gone, either for a while when recovering, or forever. Knowledge transfer is incredibly important. Things like tickets, scrum, kanban etc are used because they work for every type of person - they serve to transfer knowledge, the hierarchy in a team protects the programmers from shit they shouldn't be dealing with (that's the project manager's role, to be a shield for the team, to curate the the messages comming from "higher up" and the ones sent out by the team). The most important thing to know is that "do the needful" is about as shitty of an ask someone can pose to a programmer. People doing that don't know what they want, and instead rely on what they don't want - once you actually implement something that remotely fits what they needed. As for clients not reading specs - it might be time for someone to have "the talk" with them (obviously not you). One thing you need to know is that shitty clients can be fired too, once the development starts breaking down, the communication is arse etc.

For the people on the road - they probably took it as you trying to bail on them. But yeah, impulsivity when things are getting heated is never good, it's better to stay silent for a few seconds and then say something, rather than immediately say something that can be taken very badly.