Has the same vibes as anthropic creating a C compiler which passes 99% of compiler tests.

That last percent is really important. At least that last percent are some really specific edge cases right?

Description:
When compiling the following code with CCC using -std=c23:

bool is_even(int number) {
   return number % 2 == 0;
}

the compiler fails to compile due to bool, true, and false being unrecognized. The same code compiles correctly with GCC and Clang in C23 mode.

Source

Well fuck.

The error is ~1/log(x), for anyone interested.

Reminds me of this xkcd:

https://xkcd.com/221/

LLMs belong to the same category. Seemingly right, but not really right.

My favorite part of this is that they test it up to 99999 and we see that it fails for 99991, so that means somewhere in the test they actually implemented a properly working function.

you can increase its accuracy by changing the parameter type to long

I have seen that algorithm before. It's also the implementation of an is_gay(Image i) algorithm with around 90% accuracy.

I'm struggling to follow the code here. I'm guessing it's C++ (which I'm very unfamiliar with)

bool is_prime(int x) {
    return false;
}

Wouldn't this just always return false regardless of x (which I presume is half the joke)? Why is it that when it's tested up to 99999, it has a roughly 95% success rate then?

If you scaled it based on the size of the integer you could get that up to 99.9% test accuracy. Like if it's less than 10 give it 50% odds of returning false, if it's under 50 give it 10% odds, otherwise return false.

It has actually 100% accuracy

If you think this is bad and not nearly enough accuracy to be called correct, AI is much worse than this.

It's not just wrong a lot of times or hallucinates but you can't pinpoint why or how it produces the result and if you keep putting the same data in, the output may still vary.

100% of the time, baby =)

Pssh, mine uses a random number generator for odd numbers to return true 4% of the time to achieve higher accuracy and a bettor LLM metaphor