There is an arcade with AI-powered games, one of which is Pong. A neural network is trained on each game; the more games it plays, the better it becomes at Pong. After enough games, it gains a superhuman ability and can win every game. The optimization function of the model is to win as many games as possible.
However, after some time, the owner of the Arcade notices a peculiar behavior. The model that first won every game now started to lose some games, but the number of games played increased. The total number of wins increased.
Kids that come to the arcade to play get quickly discouraged if they lose 100% of the time, but if they sometimes get to win, they stay longer and play more games. The model has learned a truth about reality beyond the game’s mechanics. Remember that the information that the model has access to is the movements of the arcade’s joystick and buttons.
The model does not know (let’s think of it as a person for a moment since it can help grasp the idea) that the kids exist, and even if it did, it would not be able to comprehend it. The model does not know why losing sometimes helps deliver on its optimization function; all it knows is that it does.
The model is a metaphor for us. The probability that we see reality as it is is 0% (Mark, Marion, Hoffman, 2010). What we perceive through our senses is the equivalent of joystick movements and button presses. They offer a connection to reality, but they tell us nothing of its true nature; even if they did, we would likely be unable to comprehend it.
Reference
Mark, J., Marion, B., Hoffman, D., 2010. Natural selection and veridical perceptions. Journal of Theoretical Biology 266 (2010) 504–515