Mamdani to kill the NYC AI chatbot caught telling businesses to break the law

3 hours ago (themarkup.org)

Why did NYC release it in the first place? Did they not QA it?

Or was it perhaps one of those cases where they found issues, but the only way to really know for sure that the deleterious impact is significant enough by pushing it to prod?

  • >Why did NYC release it in the first place? Did they not QA it? How do you QA black box non-deterministic system? I'm not being facetious, seriously asking.

  • QA efforts can whack-a-mole some issues, but the mismatch of problem and solution is inherent in any situation in which a generator of plausible-sounding text gets pointed at an area where correctness matters.

  • Why do you think OpenAI let a red team loose on GPT-5 for six months before releasing it to the public?

  • > Why did NYC release it in the first place?

    Perhaps a big fat check was involved.

> The bot, built using Microsoft’s cloud computing platform

When is the last time there was positive news involving Microsoft? This bot could've easily been on AWS or GCP but I find it hilarious that here they are, getting dragged yet again

Being in and around the NYC area, while also knowing plenty of small businesses, I'm glad Mamdani killed this bot. Telling bosses to steal tips from their employees is run-of-the-mill corruption and common over here. The vibe for businesses is that everyone has to be exploiting someone else or have a schtick. If you were to talk about morals, you would be ridiculed. Most lawyers wouldn't even prosecute small businesses for this. It's probably why the agent was put into production, the level of business ethics in NYC is cartoonishly evil.

We’ll likely see a lot of these AI pet projects get axed in the coming year or two… especially things rushed out in the early phases of the AI bubble when folks were desperate to appear to be using AI.

> A spokesperson for the mayor, Dora Pekec, confirmed in a text message that the new administration plans to take down the chatbot. She said a member of the Mamdani transition team had seen reporting on the bot from The Markup and THE CITY and presented it to the mayor as a possible place to save funds.

Journalism works.