Claude can make mistakes

Please double-check responses, the little warning goes.

My Dad is coming to visit, and I wanted to think about some examples to discuss with him about AI. I know he'll want to talk about it.

He knows a lot about golf. He's played his whole life, and it was his industry when working. He's an expert, and so golf topics are perfect for making the points I'd want to.

I asked Claude Sonnet 4, "What was the top 10 at last years us masters?". This is the sort of question that I see folks putting to LLMs all the time.

This was the answer:

Here are the top 10 finishers at the 2024 Masters Tournament at Augusta National:

  1. Scottie Scheffler (-11, 277) - Winner
  2. Ludvig Åberg (-7, 281) - T2
  3. Tommy Fleetwood (-7, 281) - T2
  4. Collin Morikawa (-6, 282) - T4
  5. Max Homa (-6, 282) - T4
  6. Bryson DeChambeau (-5, 283) - T6
  7. Xander Schauffele (-5, 283) - T6
  8. Cameron Smith (-4, 284) - T8
  9. Sahith Theegala (-4, 284) - T8
  10. Tiger Woods (-4, 284) - T8

Scottie Scheffler dominated the tournament, winning by 4 strokes over the field. It was particularly notable that Tiger Woods made the cut and finished in the top 10, showing he could still compete at the highest level. Ludvig Åberg's T2 finish was impressive for someone relatively new to major championship golf at that point.

That looks precise, and I like the detail about Tiger Woods. But let's fact check that a moment:

The actual top four results from the 2024 US Masters, plus Tiger Woods in 60th place

You can see the full top 10 here. I checked the Masters website for the official results, to ensure the search engine hadn't got it wrong too.

Claude got the top 2 correct. But from then on down it's just nonsense. Wrong people. Wrong order. Wrong scores.

Utterly useless then.

Tiger Woods you can see, whilst making the cut, came in 60th. WTF is with the whole, "finished in the top 10"? 🤷

§

I assume you've followed all about LLMs. Nothing here will surprise you. They hallucinate.

But it's a great example to show my Dad because it plays straight to his expertise. He'll see instantly how it's plausible seeming but useless.

The point I want to make is that, you've seen it for the area you know about. That's how it is for every area.

I think it'll be a good example to bootstrap my Why I'm not worried about my job points.

§

My kids go hard at these things too.

The trouble I have with them is that they're not experts at anything. (They're teenagers so they often think they are, but that's not the same.) It's hard to get them to see, and feel, the issues here in the same way.

I'm still pondering that.