Model AI Governance Framework for Agentic AI (Version 1.5)

Source: Infocomm Media Development Authority (IMDA), Singapore; Government framework · 2026

Executive Summary Agentic AI is the next evolution of AI, holding transformative potential for users and businesses. Compared to generative AI, AI agents can take actions, adapt to new information, and interact with other agents and systems to complete tasks on behalf of humans. While use cases are rapidly evolving, agents are already transforming the workplace through coding assistants, customer service agents, and automating enterprise productivity workflows. These greater capabilities also bring forth new risks. Agents’ access to sensitive data and ability to make changes to their environment, such as updating a customer database or making a payment, are double-edged swords. As we move towards deploying multiple agents with complex interactions, outcomes also become more unpredictable. Humans must remain accountable and properly manage these risks. While existing governance principles for trusted AI such as transparency, accountability and fairness continue to apply, they need to be translated in practice for agents. Additionally, meaningful human control and oversight need to be integrated into the agentic AI lifecycle. Nevertheless, a balance needs to be struck as continuous human oversight over all agent workflows becomes impractical at scale. The Model AI Governance Framework (MGF) for Agentic AI gives organisations a structured overview of the risks of agentic AI and emerging best practices in managing these risks. If risks are properly managed, organisations can adopt agentic AI with greater confidence. The MGF is targeted at organisations looking to deploy agentic AI, whether by developing AI agents in-house or using third-party agentic solutions. Building on our previous model governance frameworks, we have outlined key considerations for organisations in four areas for agentic AI:

Introduction. There is no consensus on what defines an AI agent, but there are certain common features – agents usually possess some degree of independent planning, decision-making, and action-taking (e.g. searching the web or creating files) over multiple steps to achieve a user-defined goal.1 Agentic AI systems are software systems consisting of one or multiple AI agents that may operate individually or collaboratively. In this framework, we focus on agents built on generative AI models, which are increasingly being adopted. Generally, such agents use a small, large, or multimodal large language model (SLM, LLM, or MLLM) as its brain to make decisions and complete tasks. However, it is worth noting that software agents are not a new concept and other types of agents exist, such as those which use deterministic rules, or other neural networks, to make decisions.2 1.1.1 Core components of an agent 1. Model: an SLM, LLM or MLLM that serves as the central reasoning and planning engine, or the “brain” of the agent. It processes instructions, interprets user inputs, and generates contextually appropriate responses. 2. Instructions: Natural language commands that define an agent's role, capabilities, and behavioural constraints e.g. a system prompt for an LLM. 3. Memory: Information that is stored and accessible to the LLM, either in short or long-term storage.

4. Planning and reasoning: The model is usually trained to reason and plan, meaning that it can output files and databases, controlling devices, or performing transactions. The model calls tools to complete a task. Agents themselves can also be called as tools, e.g. one supervisor agent invokes another specialist agent for a certain task. 6. Protocols: Standardised ways for agents to communicate with tools and other agents. For example, 7. Controls: Controls limit the agent’s action-space and autonomy. While there are many types of a. Access controls: These limit what an agent is allowed to see, use or change, including restricting access to sensitive data, tools and systems. b. Guardrails: Guardrails monitor and constrain an agent’s behaviour before, during or after it acts. It can help detect unsafe instructions, policy violations, or actions that appear inconsistent with user intent. c. Human approvals: Requirements for a human to review or approve agent actions. 8. Log ging and monitoring : Records agent actions, decisions, and interactions across all components to enable monitoring, debugging, and accountability.

4 See Anthropic, Model Context Protocol and Google, Agent2Agent Protocol. 5 See OpenAI, Agentic Commerce Protocol, Alipay, Agentic Mobile Protocol, and Google, Universal Commerce Protocol. 6 More information on this can be found in 2.3 Implement technical controls and processes.

1.1.2 Multi-agent setups In an agentic system, it is common for multiple agents to be set up to work together. This allows each agent to specialise in a certain function or task and/or work in parallel. Having multiple agents specialising in different tasks also means that each agent’s tools and permissions can be separately scoped and defined, compared to a single agent with access to many tools. Three simple design patterns for multi-agent systems are:7 Action-space (or authority, capabilities) Range of actions the agent can take, including transactions it can execute, determined by the tools it is allowed to use and permissions on those tools Autonomy (or decision-making) Degree to which an agent can decide how to act towards a goal, such as by defining the steps to be taken, determined by its instructions and level of human involvement Action - space An agent’s action-space mainly depends on the tools it has access to, which can affect: rather than write to and modify data within the system. An emerging modality of agentic AI is a computer use agent, whose primary tool is access to a computer and browser. This means that it can take any action that a human can take with a computer and browser (e.g. scrolling, clicking, typing) without having to rely on specifically defined tools and APIs. This significantly increases what the agent can access and do.10 Autonomy An agent’s autonomy mainly depends on its instructions and the level of human involvement in the agentic system. In enterprise deployments, most agentic implementations are hybrid systems combining deterministic rules with some autonomy. In terms of instructions, an agent can be given differing level of instructions:

• Detailed instructions and SOP: An agent instructed to follow a detailed SOP to complete a task would be limited in the decisions it can make at each stage. • Using its own judgment: An agent instructed to use its own judgment to complete a task would have more freedom to define its plan and workflow. Another relevant factor is the level of human involvement.

Method. 2 Model AI Governance Framework for Agentic AI 1. Assess and bound the risks upfront 2. Make humans meaningfully accountable 3. Implement technical controls and processes 4. Enable end - user responsibility 2.1 Assess and bound the risks upfront Agents bring new risks, especially in their access to sensitive data and ability to change their environment through action-taking. Their adaptive, autonomous and multi-step nature also increases the potential for unexpected actions, emergent risks and cascading impacts. Organisations should consider these new dimensions through:

• Determining suitable use cases for agent deployment by considering agent-specific factors that can affect the likelihood and impact of the risk. • Design choices to bound the risks upfront by applying limits on agent’s access to tools and systems and defining a robust identity and permissions framework. 2.1.1 Determine suitable use cases for agent deployment 2.1.2 Bound risks through design by defining agents limits and permissions 2.2 Make humans meaningfully accountable The organisations that deploy agents and the humans who oversee them remain accountable for the agents’ actions. But it can be challenging to fulfil this accountability when agent actions emerge dynamically and adaptively from interactions instead of fixed logic. Multiple stakeholders may also be involved in different parts of the agent lifecycle, diffusing accountability. Finally, automation bias, or the tendency to over-trust an automated system, especially when it has performed reliably in the past, becomes a bigger concern as humans supervise increasingly capable agents. To address these challenges to human accountability, organisations should consider:

As deployers, organisations and humans remain accountable for the decisions and actions of agents. However, as with AI, the value chain for agentic AI involves multiple actors. Organisations should consider the allocation of responsibility both within their organisation, and vis-à-vis other organisations along the value chain.

Setting up a system for effective human supervision Organisations should define significant checkpoints or action boundaries that require human approval, especially before sensitive actions are executed. This can include:26 • High-stakes actions and decisions e.g. editing of sensitive data, final decisions in high-risk domains (such as healthcare or legal), actions that may trigger liability • Irreversible actions e.g. permanently deleting data, sending communications, making payments • Outlier or atypical behaviour e.g. when agent accesses a system or database outside of its work scope, when agent selects a delivery route that is twice as long as the median distance. • User-defined e.g. agents may act on behalf of users who have different risk appetites. Beyond organisation-defined boundaries, users may be given the option to define their own boundaries e.g. requiring approval for purchases above a certain amount. Apart from considering when approvals are required, organisations should also consider what form approvals should take. These considerations include:

• Keep approval requests contextual and digestible, while making the risk clear. When asking humans for approval, keep the request short and clear, instead of providing long logs or raw data that may be challenging to decipher. However, useful additional data, such as the associated risk with the action or a confidence score, should be included. • Consider the form of human input required. For straightforward actions such as accessing a database, the human user can simply approve or reject. For more complex cases, such as reviewing an agent’s plan before execution, it may be more productive for the human to edit the plan before giving the agent the go-ahead. For high-risk actions, human users can be required to provide an additional written justification before approving or rejecting. Organisations should implement measures to ensure continued effectiveness of human oversight, particularly as humans remain susceptible to alert fatigue and automation bias and may be influenced by anthropomorphic design. These measures can include: