My current (possibly wrong) understanding is that jedi keep the planet with the crystals secret, so that not just anyone can go make a saber. On top of that, the Force allows a user to kinda “extend” their perception, which allows for things like reflecting blaster shots, as well as being hyper aware of their personal space during saber duels. So a trained jedi can “see” things like how far a blade is from hitting them, even if they can’t actually see, and that determines how they move.

So it’s more that taking one would be like robbing a special forces soldier, and playing with one would most likely kill or dismember you.