At the beginning of the week, one of the developers using the famous codes backed by artificial intelligence “Cursor” noticed a strange behavior, as the login from another device immediately led to the out of the previous device, which disrupts the usual workflow of many programmers who move between more than one device.
When the user communicated with the support service, he received a response from an employee named “Sam”, he explained to him that this behavior was caused by a new policy, but there was no policy of this kind, and “Sam” was not human, but rather a robot supported by artificial intelligence invented the policy on its own, which sparked a wave of anger and threats to cancel the contributions.
This incident is the latest example of what is known as “hallucinations” that result from artificial intelligence models, a phenomenon in which responses that appear reasonable but incorrect, and large linguistic models often prefer to provide confident and convincing answers instead of recognizing not to confirm even if it is necessary to form baseless information.
For companies that use artificial intelligence in the tasks of communicating with customers without sufficient human control, the results may be direct and costly such as customer discontent, loss of confidence, and the possibility of canceling subscriptions, as happened in the case of “Cursor”.
Halosa risks in the business environment
This incident is remembered by a similar incident that occurred in February 2024 when “Air Canada” was forced to adhere to a recovery policy invented by her chat robot, and at that time “Jake Movat” continues with the company’s support service after the death of his grandmother and told him the robot can reserve a ticket at an ordinary price and subsequently apply for compensation for humanitarian cases, and when the company refused the recovery request defended It itself is that “the robot is a separate legal entity responsible for its own actions,” but a Canadian court rejected this claim, and confirmed that companies bear the responsibility of the information provided by its artificial intelligence tools.
Unlike “Air Canada”, “Cursor” confessed to her mistake and took steps for correction, as the founding partner of the company “Michael Trol” apologized for “Hacker News”, explaining that the user had received the payment of the amount paid, and that the problem arose due to a technical amendment that was aimed at enhancing the safety of the sessions, but it unintentionally caused the problems of logging out to some users.
“Trol” added that all artificial intelligence responses used in e -mail support are now clearly a mechanism, and we use artificial intelligence responses as a primary filter for responses.
However, many questions remained among users about transparency, especially after it became clear that many of those who interacted with “Sam” believed to be a human employee, and one of them wrote on “Hacker News”: “Naming the robot in the name of humanity and not clarifying his true identity is clearly deceptive.”
Although “Cursor” dealt with the technical error, the incident revealed the risks of using artificial intelligence in direct interaction with customers without adequate guarantees or an appropriate level of disclosure, and for a company that promotes productive tools that depend on artificial intelligence and target developers, the occurrence of an error of this type by its support system is a embarrassing fall that is difficult to justify.
One of the users mocked, “It is ironic that some are trying hard to reduce the problem of hallucinations, then a company that relies on this narration comes to directly damage it.”
