Checklist 437: Wrong Numbers from AI

AI Hallucinations Move From Coffee Shops to Cybercrime Scams

A lighthearted story about a coffee order gone wrong has given way to a far more serious warning about the dangers of trusting artificial intelligence (AI) outputs without question.

The anecdote—first shared on Mastodon—involved a customer who claimed he had asked ChatGPT to place a mobile coffee order. When the drink was nowhere to be found, the barista gently explained that no such order existed. The punchline: ChatGPT had simply “said” the order would be ready, when in fact it wasn’t.

While the coffee shop incident was only an inconvenience, the podcast warns that the same misplaced trust in AI is being weaponized by scammers.

Prompt Injection & Fake Phone Numbers

According to ZDNet, cybercriminals are exploiting Google’s AI Overviews through prompt injection attacks, a method defined by Palo Alto Networks as feeding deceptive text into a large language model (LLM) to manipulate its output. This can include planting fake customer service phone numbers in search result.

Unlike traditional search results—where users see multiple options and can cross-check—AI summaries often deliver a single authoritative answer, making users more vulnerable.

Real-World Scams Emerging

Several cases illustrate the risk:

• Alex Rivlin, CEO of D.C.-area real estate firm Rivlin Group, told The Washington Post that he was duped into calling a fake number for Royal Caribbean, supplied by Google’s AI Overview. The scammers had detailed knowledge of cruise operations, convincing him to hand over his credit card information. Fraudulent charges followed.
• DigitalTrends reported another case where a shopper, missing items from an online order, turned to Google for customer service help. Google’s AI Overview offered a phone number—even though the retailer only provides chat support. The supposed “agent” requested a WhatsApp number and attempted a screen-sharing scam before the user hung up.

Tech Companies Respond—But Slowly

Both Google and OpenAI acknowledge the problem:

• Google told The Washington Post it has “taken action” on fake phone numbers and is working on “broader improvements.”
• OpenAI told ZDNet that many fake-number pages have been removed but cautioned that updates “can take time.”

Critics argue that assurances such as these minimize the risks faced by victims already scammed.

Staying Safe Online

Experts advise skepticism when dealing with AI-generated contact information. ZDNet suggests:

• Avoid dialing numbers provided by AI tools.
• Add “-AI” to search queries to force more traditional results.
• Whenever possible, visit a company’s official website directly for verified contact details.

AI tools are not yet reliable intermediaries for customer service or transactions. Users should double-check information and resist the temptation to treat AI outputs as fact.

Apple Pushes Surprise Updates, Including Critical Security Fixes

Apple has issued a string of unexpected operating system updates over the past two weeks, some focused on features, others on urgent security.

Health Features Quietly Restored

Last week, Apple rolled out iOS 18.6.1 and watchOS 11.6.1—updates that surprised many given the company’s usual September release cycle for major new versions. Unlike most “dot-dot-one” patches, these were not about security. Instead, they re-enabled Blood Oxygen readings on the Apple Watch Series 9, Series 10, and Ultra 2. The feature had been disabled due to ongoing legal battles but has now returned, marking a significant update for affected watch owners.

This Week: Critical Security Updates

Far more urgent were the releases this week of iOS 18.6.2, iPadOS 18.6.2, and macOS Sequoia 15.6.1. According to MacRumors, these updates address a zero-day vulnerability believed to have been actively exploited.

Apple’s release notes say the flaw may have been leveraged in “an extremely sophisticated attack against specific targeted individuals,” language that typically signals nation-state-level cyber operations targeting dissidents, journalists, or activists.

The vulnerability resides in ImageIO, with Apple warning that “processing a malicious image file may result in memory corruption.” The company says it resolved the issue through improved bounds checking.

Extended Fixes for Older Systems

Apple also pushed security patches to older versions of its platforms, releasing:

• iPadOS 17.7.10
• macOS Sonoma 14.7.8
• macOS Ventura 13.7.8

This broader rollout underscores the seriousness of the flaw, ensuring users on legacy systems remain protected.

We advises users to keep devices set to update automatically and stresses the importance of applying these patches quickly. While most users may not be direct targets of such attacks, the organization warns that staying current on security updates is essential for everyone.

Even if you’re not a journalist or dissident, Apple’s latest updates are critical. Users should update immediately to safeguard their devices against potential exploitation.