Artificial intelligence is taking on increasingly complex tasks—but who remains responsible when AI systems make decisions or trigger actions in production environments? This is exactly the question explored by Dr. Sönke Magnussen and Mahbouba Gharbi, curators of the CPSA-Advanced-Level module SWARC4AI – Software Architecture for AI Systems. In this interview, they explain why AI agents cannot assume responsibility, the risks posed by fully autonomous agents, and how software architectures can ensure governance, security, and accountability in AI-driven systems.
- AI agents are taking on increasingly complex tasks. Why is it problematic to assume that they can replace human responsibility?
LLMs generate probabilities, but they do not assume responsibility in a legal, professional, or organizational sense. They can justify decisions, but they cannot be held accountable or fulfill compliance requirements. Responsibility always remains with people and organizations. Therefore, software architectures must clearly define where AI provides support and where critical decisions require human review or approval.
- You mention that LLM-based agents can quickly become a “single point of failure.” Why is that, and what risks does this create for organizations?
Many agent architectures combine understanding, decision-making, and execution within a single component. As a result, errors, manipulation, or misinterpretations can immediately affect business processes and production systems. This becomes particularly critical when agents have extensive permissions or can automatically invoke tools and services. A single mistake can lead to financial losses, compliance violations, or security incidents.
- Many teams currently rely on guardrails or prompt hardening. Why do you believe these measures alone are not enough to make AI systems truly secure and controllable?
Guardrails and prompt hardening are important security mechanisms for AI systems, but they operate primarily at a local and heuristic level. They evaluate individual inputs, outputs, or actions, but they cannot guarantee that a complex business process is executed correctly and in compliance with policies across multiple steps. Requirements such as four-eyes approval principles, approval workflows, separation of duties, or regulatory approval chains require technically enforceable process controls and traceable state transitions. Guardrails reduce risk and improve system robustness, but they cannot replace process governance. For business-critical or regulated processes, they are essential—but not sufficient on their own.
- Which architectural approaches help restore responsibility and control in AI systems, and how can they be implemented in practice?
Proven architectural patterns include Human Approval Gates, Policy Decision Points, least-privilege access, and comprehensive audit trails. In these architectures, the LLM is responsible for generating recommendations, while rules, approvals, and permissions are enforced by independent components—typically implemented as conventional software. In practice, this means that production actions are executed through controlled workflows, rule engines, or managed services. Governance is therefore enforced technically rather than merely described in prompts.
- You describe architectures in which AI is not responsible for making the final decisions. What does such an architecture look like, and what advantages does it offer compared to fully autonomous agents?
A robust architecture separates understanding, planning, and execution into distinct responsibilities. The LLM analyzes the situation, conventional software or planning components create a policy-compliant execution plan, and controlled systems perform the actual actions. This makes critical decisions transparent, auditable, and reproducible. The strengths of AI are preserved while the risks associated with uncontrolled autonomy are significantly reduced.