▲ 973 ▼ Managers (thelemmy.club) submitted 4 months ago* by inari@piefed.zip to c/whitepeopletwitter@sh.itjust.works 180 comments fedilink hide all child comments
[–] Evotech@lemmy.world 1 point 4 months ago (9 children) And then add 200k context on top And then add hundred of users needing to do things in paralell permalink fedilink source parent hideshow 9 child comments replies: [–] lime@feddit.nu 1 point 4 months ago nobody said anything about it being a large company :P anyway, seems the framework is hampered by a slow gpu so the memory issues are apparently moot. permalink fedilink source parent [–] boonhet@sopuli.xyz 1 point 4 months ago (7 children) If it's a large enough company to have hundreds of users, it can afford several beefy machines tbh permalink fedilink source parent hideshow 7 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (6 children) It's a capex and that type of hardware needs to be replaced every 3 years minimum and you need people to set it up and maintain a cluster. And it's not straight forward. You are never going to get that approved without a serious business case. Claude on the other end is a opex and much easier to just try out and then build a solution on it Not saying it doesn't happen but it's not as easy as people make it sound like permalink fedilink source parent hideshow 6 child comments replies: [–] boonhet@sopuli.xyz 1 point 4 months ago (5 children) It's 3 years if you're trying to be competitive on frontier models and generally capex is preferred to opex because opex never ends I don't think anyone's building a cluster for their business right now, but one single rack after Claude gets rid of their subscription options? Might be a good deal. permalink fedilink source parent hideshow 5 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (4 children) Capex never ends either if it's hardware. Also you need opex to run it permalink fedilink source parent hideshow 4 child comments replies: [–] boonhet@sopuli.xyz 1 point 4 months ago (3 children) 400k on a DGX node starts seeming like a great deal when your employees each start using a few hundred dollars worth of Claude tokens every month. That one node can handle a lot of users depending on the model used. It's an expense once every maybe 5 or 6 years in reality and you don't need to hire new people, you just give your existing sysadmins some extra work. They'll complain, but they'll still do it. Of course the sensible alternative is to use a decent model off openrouter for peanuts but then you're sending all your sensitive business secrets to China which is even worse than sharing them with a US AI company. And people WILL be sharing secrets lol permalink fedilink source parent hideshow 3 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (2 children) If only it was that simple permalink fedilink source parent hideshow 2 child comments replies: [–] boonhet@sopuli.xyz 1 point 3 months ago (1 child) You don't have to run Claude Opus for it to be useful lol permalink fedilink source parent hideshow 1 child comment replies: [–] Evotech@lemmy.world 1 point 3 months ago It's always going to be second rate. And you'll have to defend that permalink fedilink source parent
[–] lime@feddit.nu 1 point 4 months ago nobody said anything about it being a large company :P anyway, seems the framework is hampered by a slow gpu so the memory issues are apparently moot. permalink fedilink source parent
[–] boonhet@sopuli.xyz 1 point 4 months ago (7 children) If it's a large enough company to have hundreds of users, it can afford several beefy machines tbh permalink fedilink source parent hideshow 7 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (6 children) It's a capex and that type of hardware needs to be replaced every 3 years minimum and you need people to set it up and maintain a cluster. And it's not straight forward. You are never going to get that approved without a serious business case. Claude on the other end is a opex and much easier to just try out and then build a solution on it Not saying it doesn't happen but it's not as easy as people make it sound like permalink fedilink source parent hideshow 6 child comments replies: [–] boonhet@sopuli.xyz 1 point 4 months ago (5 children) It's 3 years if you're trying to be competitive on frontier models and generally capex is preferred to opex because opex never ends I don't think anyone's building a cluster for their business right now, but one single rack after Claude gets rid of their subscription options? Might be a good deal. permalink fedilink source parent hideshow 5 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (4 children) Capex never ends either if it's hardware. Also you need opex to run it permalink fedilink source parent hideshow 4 child comments replies: [–] boonhet@sopuli.xyz 1 point 4 months ago (3 children) 400k on a DGX node starts seeming like a great deal when your employees each start using a few hundred dollars worth of Claude tokens every month. That one node can handle a lot of users depending on the model used. It's an expense once every maybe 5 or 6 years in reality and you don't need to hire new people, you just give your existing sysadmins some extra work. They'll complain, but they'll still do it. Of course the sensible alternative is to use a decent model off openrouter for peanuts but then you're sending all your sensitive business secrets to China which is even worse than sharing them with a US AI company. And people WILL be sharing secrets lol permalink fedilink source parent hideshow 3 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (2 children) If only it was that simple permalink fedilink source parent hideshow 2 child comments replies: [–] boonhet@sopuli.xyz 1 point 3 months ago (1 child) You don't have to run Claude Opus for it to be useful lol permalink fedilink source parent hideshow 1 child comment replies: [–] Evotech@lemmy.world 1 point 3 months ago It's always going to be second rate. And you'll have to defend that permalink fedilink source parent
[–] Evotech@lemmy.world 1 point 4 months ago (6 children) It's a capex and that type of hardware needs to be replaced every 3 years minimum and you need people to set it up and maintain a cluster. And it's not straight forward. You are never going to get that approved without a serious business case. Claude on the other end is a opex and much easier to just try out and then build a solution on it Not saying it doesn't happen but it's not as easy as people make it sound like permalink fedilink source parent hideshow 6 child comments replies: [–] boonhet@sopuli.xyz 1 point 4 months ago (5 children) It's 3 years if you're trying to be competitive on frontier models and generally capex is preferred to opex because opex never ends I don't think anyone's building a cluster for their business right now, but one single rack after Claude gets rid of their subscription options? Might be a good deal. permalink fedilink source parent hideshow 5 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (4 children) Capex never ends either if it's hardware. Also you need opex to run it permalink fedilink source parent hideshow 4 child comments replies: [–] boonhet@sopuli.xyz 1 point 4 months ago (3 children) 400k on a DGX node starts seeming like a great deal when your employees each start using a few hundred dollars worth of Claude tokens every month. That one node can handle a lot of users depending on the model used. It's an expense once every maybe 5 or 6 years in reality and you don't need to hire new people, you just give your existing sysadmins some extra work. They'll complain, but they'll still do it. Of course the sensible alternative is to use a decent model off openrouter for peanuts but then you're sending all your sensitive business secrets to China which is even worse than sharing them with a US AI company. And people WILL be sharing secrets lol permalink fedilink source parent hideshow 3 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (2 children) If only it was that simple permalink fedilink source parent hideshow 2 child comments replies: [–] boonhet@sopuli.xyz 1 point 3 months ago (1 child) You don't have to run Claude Opus for it to be useful lol permalink fedilink source parent hideshow 1 child comment replies: [–] Evotech@lemmy.world 1 point 3 months ago It's always going to be second rate. And you'll have to defend that permalink fedilink source parent
[–] boonhet@sopuli.xyz 1 point 4 months ago (5 children) It's 3 years if you're trying to be competitive on frontier models and generally capex is preferred to opex because opex never ends I don't think anyone's building a cluster for their business right now, but one single rack after Claude gets rid of their subscription options? Might be a good deal. permalink fedilink source parent hideshow 5 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (4 children) Capex never ends either if it's hardware. Also you need opex to run it permalink fedilink source parent hideshow 4 child comments replies: [–] boonhet@sopuli.xyz 1 point 4 months ago (3 children) 400k on a DGX node starts seeming like a great deal when your employees each start using a few hundred dollars worth of Claude tokens every month. That one node can handle a lot of users depending on the model used. It's an expense once every maybe 5 or 6 years in reality and you don't need to hire new people, you just give your existing sysadmins some extra work. They'll complain, but they'll still do it. Of course the sensible alternative is to use a decent model off openrouter for peanuts but then you're sending all your sensitive business secrets to China which is even worse than sharing them with a US AI company. And people WILL be sharing secrets lol permalink fedilink source parent hideshow 3 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (2 children) If only it was that simple permalink fedilink source parent hideshow 2 child comments replies: [–] boonhet@sopuli.xyz 1 point 3 months ago (1 child) You don't have to run Claude Opus for it to be useful lol permalink fedilink source parent hideshow 1 child comment replies: [–] Evotech@lemmy.world 1 point 3 months ago It's always going to be second rate. And you'll have to defend that permalink fedilink source parent
[–] Evotech@lemmy.world 1 point 4 months ago (4 children) Capex never ends either if it's hardware. Also you need opex to run it permalink fedilink source parent hideshow 4 child comments replies: [–] boonhet@sopuli.xyz 1 point 4 months ago (3 children) 400k on a DGX node starts seeming like a great deal when your employees each start using a few hundred dollars worth of Claude tokens every month. That one node can handle a lot of users depending on the model used. It's an expense once every maybe 5 or 6 years in reality and you don't need to hire new people, you just give your existing sysadmins some extra work. They'll complain, but they'll still do it. Of course the sensible alternative is to use a decent model off openrouter for peanuts but then you're sending all your sensitive business secrets to China which is even worse than sharing them with a US AI company. And people WILL be sharing secrets lol permalink fedilink source parent hideshow 3 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (2 children) If only it was that simple permalink fedilink source parent hideshow 2 child comments replies: [–] boonhet@sopuli.xyz 1 point 3 months ago (1 child) You don't have to run Claude Opus for it to be useful lol permalink fedilink source parent hideshow 1 child comment replies: [–] Evotech@lemmy.world 1 point 3 months ago It's always going to be second rate. And you'll have to defend that permalink fedilink source parent
[–] boonhet@sopuli.xyz 1 point 4 months ago (3 children) 400k on a DGX node starts seeming like a great deal when your employees each start using a few hundred dollars worth of Claude tokens every month. That one node can handle a lot of users depending on the model used. It's an expense once every maybe 5 or 6 years in reality and you don't need to hire new people, you just give your existing sysadmins some extra work. They'll complain, but they'll still do it. Of course the sensible alternative is to use a decent model off openrouter for peanuts but then you're sending all your sensitive business secrets to China which is even worse than sharing them with a US AI company. And people WILL be sharing secrets lol permalink fedilink source parent hideshow 3 child comments replies: [–] Evotech@lemmy.world 1 point 4 months ago (2 children) If only it was that simple permalink fedilink source parent hideshow 2 child comments replies: [–] boonhet@sopuli.xyz 1 point 3 months ago (1 child) You don't have to run Claude Opus for it to be useful lol permalink fedilink source parent hideshow 1 child comment replies: [–] Evotech@lemmy.world 1 point 3 months ago It's always going to be second rate. And you'll have to defend that permalink fedilink source parent
[–] Evotech@lemmy.world 1 point 4 months ago (2 children) If only it was that simple permalink fedilink source parent hideshow 2 child comments replies: [–] boonhet@sopuli.xyz 1 point 3 months ago (1 child) You don't have to run Claude Opus for it to be useful lol permalink fedilink source parent hideshow 1 child comment replies: [–] Evotech@lemmy.world 1 point 3 months ago It's always going to be second rate. And you'll have to defend that permalink fedilink source parent
[–] boonhet@sopuli.xyz 1 point 3 months ago (1 child) You don't have to run Claude Opus for it to be useful lol permalink fedilink source parent hideshow 1 child comment replies: [–] Evotech@lemmy.world 1 point 3 months ago It's always going to be second rate. And you'll have to defend that permalink fedilink source parent
[–] Evotech@lemmy.world 1 point 3 months ago It's always going to be second rate. And you'll have to defend that permalink fedilink source parent