That's actually a pretty interesting question. As Feinberg was using it, almost certainly transsexual was referring specifically to men and women, and not nonbinary or third-gender people, nor even intersex people. The language at the time for transsexual was fairly narrow. The thing that makes it so interesting is that, because the language was narrow, there would have been plenty of people who don't fall into the "binary transsexual" category that would have related to themselves and used the identifier of transsexual. So yes, there would have been "nonbinary" transsexual people.

It's impossible to apply concepts and social relations across time, so we can never know: how many people who considered themselves transsexual at the time would have used different language if they had access to it? For the same reason, many people who use terms like nonbinary now would be more likely to have used labels like dyke, butch, and fairy moreso than transsexual.

Not long after this book was written, there was a larger linguistic shift to a "transgender" framework. This is partially due to the success of Feinberg and other's like hir, as well as the rise of queer theory in the American academia. The linguistic linking of a multitude of gender transgressive identities under the single identity of transgender became a standard shorthand, and gained traction in the mainstream. This also functioned as a way to placate bio-essentialist viewpoints, by theorizing a split between sex and gender, and placing transgender as the transgression of socially constructed gender, while not challenging the biological sex.

This had the positive in that it helped cement a larger community identity that was easily recognizable and understandable to the mainstream, which can be helpful in organizing for recognition and rights. However it also had the negative effect of creating a universalizing narrative of what transgender was, which also forced many people originally intended to be encompassed under the term transgender into the position of cisgender. It created, in effect, a new binary: cisgender and transgender, which pushed some transgender people into the cisgender camp, and erased a lot of the nuance and complex identities that existed in favour of something more easily legible. This is also where a lot of the tension between the language of transgender and transsexual in current discourse stems. Transsexual was being replaced, transgender was being flattened to a more "inclusive" synonym for transsexual (not inclusive to cisgender people who were previously transgender, though), and transsexual as a term with its own specificity was being conflated with medicalism and robbed of its validity.

Currently you'll see arguments that transsexual is a reinforcing of bioessentialist or medicalist ideas of sex and gender, but I would argue it's the opposite. By conceding that transsexuality is inherently about medicalist interventionism, you accept that gender is the social construct and sex is the biological reality--but of course, sex is just as much a socially constructed category! And the difference between sex and gender is illegible in most non-English languages to begin with.

Anyway, bit of a tangent because the emergence of new linguistic structures at times of political upheaval are always fraught, and have complex histories. But as Feinberg was using it: transgender was meant to be a coalition of all people and identities that challenged the gender norms ("gender outlaws"), and so nonbinary identities (like hirs) would have been quite comfortable in the transgender umbrella (though usually with other local/subcultural language that was more specific), but likely would have felt just as constrained or unrepresented by transsexual identity as they would in cissexual identity.