I started this experiment thinking, "How can this replace me?" or "How can I set this up so I never have to do these tasks again?" Instead, I came away with something completely different—an approach to understanding confusing parts of our product by watching AI attempt tasks and struggle.

‍

‍

The Experiment

I asked ChatGPT Operator to perform a Salesforce deployment using Salto. Here’s the exact prompt I used:

"Using Salto, create a deployment between Developer and Partner orgs. I don’t care what metadata is deployed."

With some guidance, it navigated Salto’s UI, set up the deployment, and even reported any errors back to me. But the real insight came from watching where it struggled.

‍

My High-Level Thoughts

✅ It technically worked, which is exciting.
✅ Operator generally knew when to stop and ask for important decisions.
✅ It understood Salesforce concepts at a high level and seemed to know where to look.
❌ It was very slow and didn't replace a human doing the same tasks.
❌ It could still make mistakes if the UI isn’t clear enough.

‍

AI as a Tool for User Research

What stood out the most was not just how Operator interacted with Salto but what that revealed about our own product design. Watching an AI struggle through tasks provided an unexpected form of user research:

• Where did it hesitate? These moments revealed areas where navigation or terminology might be unclear.
• What mistakes did it make? Just like humans, an AI can be misled by confusing UI.
• Did it need unnecessary guidance? If AI repeatedly required manual intervention, those steps might confuse real users, too.

‍

What This Means for Product Design

Rather than asking, "Can AI replace humans for these tasks?" a more valuable question might be:

• How well do we guide users through critical operations?
• Could AI-driven tests help identify confusing parts of the product?
• What can we learn about our own UI and workflows by observing an AI struggle?

This experiment didn’t convince me that AI is ready to automate deployments fully, but it did make me want to use Operator as a user research tool—a way to see Salto through fresh eyes and spot areas that could be improved.

By using AI not just for automation but as a diagnostic tool, we can make our products more intuitive for both humans and AI alike.