Unless the vendor is rolling something super custom, for the communication TO the keyboard, it should use USB HID.

Start Wireshark, filter for hid, connect the KB and the first message should be a HID descriptor of the KB, look for Output Reports (it's meant from the POV of the usb master) or Feature Reports.
Though, this will probably not yield much insight - vendors love to do the easy thing, reserve opaque 32x8 bytes as a "downlink" Output communication in the Vendor Usage Page and stuff their own protocol/encoding in there.

On linux I can recommend hid-tools for working with this, in windows I believe your only solution is Wireshark.

https://www.marcusfolkesson.se/blog/hid-report-descriptors/

Happy Hacking!

E: About the already reversed software, for logitech (and more) stuff, there is piper but you will want to look into the underlying daemon libratbag, there is also solaar

I reversed engineered a keyboard for a presentation in uni. I’ll drop you an excerpt of a written review:

Resources used I learned the USB protocol from this (the relevant parts I needed). We’re thinking of including some basic understanding of the USB protocol in the slides. https://www.beyondlogic.org/usbnutshell/usb6.shtml I got an idea of what could be going on from the following link, specifically the section I’ve set. https://github.com/openrazer/openrazer/wiki/Reverse-Engineering-USB-Protocol#phase-4%E2%80%94decoding-the-protocol I deciphered the protocol using the USBHID packets that would be sent. I was highly sure it was USBHID from a pointer from another Linux community member, but this article was my third source to confirming this. https://hackaday.com/2020/04/14/reverse-engineering-an-rgb-keyboard-under-linux/ One of the sources for information to develop these procedures was from the openRGB wiki. This stream has to do with reversing using URB. I find this might be out of scope, and it would’ve been way tougher to reverse engineer with this.

Feel free to ask as needed here. Spam the requests on the software while monitoring wireshark to be sure of what is what.

The other large comment by “taaz” is also very useful and parts of which I did use while reverse engineering.

I use ckb-next to make my RGB keyboard work on Linux. Maybe you can find some ideas by looking into that project

What keyboard? There is a small chance you can rip out the chip and replace it with a teensy running qmk. That will allow you to get it to work and be able to config it in Linux.

Not sure if it's more or less work that RE'ing the communication, but it will require soldering

for a more low-level discussion for fundamentals, Ben Eater has 5 videos going over PS/2 keyboards and then USB keyboards. Here is the first video: https://www.youtube.com/watch?v=7aXbh9VUB3U

a [...] software

Nope. That's not how that word works.