2026-06 · 3 min read
When interaction becomes governance
Agentic systems don't just execute instructions. They adapt. That changes what feedback is for — and who is responsible for what the system learns.
For most of the history of software, interaction and governance were separate concerns.
Users interacted with the system. Organizations governed the outcomes. The relationship was relatively straightforward. Software executed predefined rules. If those rules needed to change, people changed them. The system itself remained largely stable.
That assumption is becoming increasingly fragile.
Agentic systems do not simply execute instructions. They interpret goals. Prioritize information. Coordinate actions. Adapt behavior. And increasingly, they do so while work is already unfolding.
This creates a subtle shift. Every correction becomes more than a correction. Every override becomes more than an override. Every clarification becomes more than a clarification. They become signals that shape future behavior.
For decades, feedback primarily improved products. Today, feedback increasingly governs systems.
That distinction matters.
A user correcting a dashboard filter rarely changed how software behaved for everyone else. A user correcting an AI-driven recommendation may influence future prioritization, classification, escalation, or delegation behavior. The interaction no longer ends with the interaction. It becomes part of the operating model.
This changes the role of design. In adaptive systems, feedback no longer helps teams understand whether the experience is working. It helps organizations regulate how systems learn.
The challenge is that learning and governance pull in opposite directions.
Systems need correction to remain aligned. People cannot spend their working lives correcting systems. Ask for feedback too often and users disengage. Ask too rarely and the system drifts. Both outcomes create the same problem: a growing gap between organizational intent and system behavior.
The interesting question is not how much feedback a system collects. It is where feedback becomes consequential.
A correction to wording is usually harmless. A correction to how risk is classified may not be. A clarification about intent may affect a single task. A clarification about escalation criteria may influence hundreds of future decisions.
Not all feedback carries the same organizational weight. Most interfaces treat it as though it does. That increasingly feels insufficient.
As systems become more adaptive, organizations will need to think less about collecting feedback and more about governing adaptation.
What is the system allowed to learn? Who can influence that learning? Which corrections remain local? Which become shared behavior? Which changes require review?
These questions sound operational. Increasingly, they are design questions. Because the answers determine how people and systems shape one another over time.
For years, interaction design focused on helping people understand software. Agentic systems introduce a second challenge: helping organizations understand how software is changing.
The future may not belong to the systems that learn the fastest. It may belong to the systems that remain governable while they learn.
Many of the ideas in this reflection were inspired by the work of Itamar Medeiros, Christopher Noessel, Lucy Suchman, Donald Norman, and the broader traditions of Human–Computer Interaction, CSCW, cybernetics, and adaptive systems research.