Why do LLMs consistently struggle with math? AI expert Karpathy explains 9.9 < 9.11

Recently, the global discussion around the seemingly simple question 'Is 9.11 greater than 9.9?' has revealed the shortcomings of large language models (LLMs) in handling basic logical problems, a phenomenon referred to as 'zigzag intelligence' or 'uneven intelligence'. Expert Andrej Karpathy pointed out that while LLMs can tackle complex tasks, they often perform poorly on certain straightforward questions, reflecting an imbalance in the models' intelligence. For example, OpenAI researcher Noam Brown found that LLMs struggle with Tic-Tac-Toe.