The AI landscape is evolving at breakneck speed. Previously, AI systems were primarily assistive and reactive, offering recommendations or performing predefined tasks when asked. Now they are entering the era of agentic AI: systems that operate autonomously, adapt in real time, and collaborate like digital colleagues.

But as AI becomes more independent, new risks emerge. So, how can we navigate this next frontier responsibly? This is a question that we at SAP do not leave to chance.

From tools to teammates

Imagine you’re buying a car. You expect it to meet all safety standards, regardless of where the component parts are built or how the car is assembled. The process behind the scenes does not change your expectation of safety. The same goes for agentic AI.

Agentic AI systems are more than tools; they are intelligent agents that plan, learn from experience, self-correct, and collaborate. They’re capable of orchestrating complex processes, making decisions, and even engaging with other agents or humans to achieve a goal. However, with this leap forward comes a new layer of complexity and risk.

Core capabilities and risks of agentic AI

Agentic AI systems bring powerful capabilities like planning, reflection, and collaboration, enabling them to tackle complex tasks autonomously. They can map strategies, learn from mistakes, use external tools, and coordinate with humans and other agents.

However, each strength introduces risks. For example, flawed planning can cause inefficiencies, reflection may reinforce unethical behavior, tool usage can lead to instability when systems interact unpredictably, and unclear collaboration can result in miscommunication and compounded errors. Balancing these capabilities with proper safeguards is essential for safe, ethical deployment.

Managing autonomy: balancing freedom with control

One of the most pressing challenges with agentic AI is managing its autonomy. Left unchecked, these systems can veer off course, misinterpret context, or introduce subtle risks without immediate detection. To address this, organizations must strike a careful balance between freedom and control.

We have learned that oversight should be calibrated according to risk. High-stakes domains like healthcare or human resources demand robust human supervision, while low-risk, routine tasks can tolerate greater autonomy. Also, continuous monitoring is essential; agentic AI systems, like any complex technology, require regular checks to ensure quality, compliance, and reliability.

A key element of this oversight is maintaining a “human in the loop” approach, where human judgment is integrated into critical decision points, ensuring that automated actions remain aligned with human values and organizational intent.

This principle has been at the heart of SAP’s ethical AI approach from the beginning, reflecting our belief that AI should augment, not replace, human decision-making. To reinforce this, SAP has introduced mandatory ethics reviews for all agentic AI use cases, ensuring that each deployment is scrutinized for ethical implications and remains aligned with our responsible AI principles.

Building transparency and accountability

Transparency is not just a buzzword; it’s a foundational requirement for building trust in agentic AI. From the outset, during the design phase, it is crucial to classify AI systems based on the complexity and risk of the tasks they perform. This classification guides decisions about the necessary safeguards and ensures that mechanisms for human intervention are integrated from the beginning.

At runtime, transparency is maintained through explainability and traceability. Developers and end-users must be able to understand what the system is doing and why. Crucially, accountability must always rest with humans or legal entities, never with the AI itself.

Rethinking governance and regulation

Despite the emergence of agentic AI, there have been no new regulations specifically crafted for it. Existing laws and frameworks such as GDPR still apply and provide a solid foundation for governance. However, what has changed is the level of technical rigor required to remain compliant and ethically sound. Organizations must now adopt more robust processes. They need to analyze use cases with greater precision, apply risk-based controls that match the potential impact of the AI system, and ensure that ethical and legal standards are upheld through enhanced design practices and ongoing testing.

Designing with human values at the center

Agentic AI cannot be an excuse for lowered standards. At SAP, the stance is unequivocal: Even in autonomous systems, AI must meet the highest ethical benchmarks. This means embedding principles such as fairness, transparency, and human agency directly into the design.

Ultimately, all users should be equipped with the tools and understanding they need to supervise and, when necessary, intervene in the system’s behavior.

Building trust in a black-box world

Trust in AI doesn’t happen by default; it must be intentionally built and continually reinforced. One of the most effective ways to do this is by giving stakeholders the right amount of information. Too much detail can be overwhelming and counterproductive while too little fosters blind trust or fear of the unknown. The key lies in communicating clearly about the system’s capabilities, risks, limitations, and appropriate use. Empowering users to critically assess the AI’s behavior – and to know when to step in – is central to creating a safe, secure, and trusted AI environment.

Rethinking KPIs in the AI-augmented workplace

As agentic systems, like our Joule Agents, begin handling more tasks, human roles will naturally evolve. To keep up with this shift, organizations need to rethink how they define and measure success. This starts with investing in change management and upskilling programs that prepare employees to work effectively alongside AI. It also requires redefining productivity metrics, moving beyond task completion to focus on how well humans and AI agents collaborate. Success should be measured by how efficiently teams harness AI to unlock new levels of insight and innovation.

Building AI that builds trust

Agentic AI is not just another phase; it is a transformation. But like any transformative technology, success depends on how it’s built, governed, and used.

At its best, agentic AI amplifies human capabilities, accelerates innovation, and helps tackle challenges once considered too complex. But it also demands a new level of diligence, oversight, and ethical reflection.

The future is not just about building smarter agents; it’s about building responsible ones.

Learn more:

• 2025 SAP Sapphire Innovation Guide
• AI Ethics Handbook, Guidelines & Global AI Ethics Steering Committee
• Learning Journey: Putting AI Ethics into Practice at SAP
• The Evolution from Chatbots to Agentic AI Systems

Walter Sun is senior vice president and head of AI at SAP.

Today at SAP Sapphire, Amazon Web Services, Inc. and SAP announced the launch of a new AI Co-Innovation Program to help partners build generative artificial intelligence applications and agents that help customers rapidly solve real-time business challenges.

Many organizations recognize generative AI’s potential to transform their business, but don’t know where to start. By combining advanced generative AI technologies with enterprise resource planning (ERP) data from critical systems, companies can unlock significant enterprise value: for example, optimizing delivery routes, anticipating potential impacts to supply chain operations, or developing precise financial outlooks.

The AI Co-Innovation Program represents the two companies’ shared vision to help partners define, build, and deploy generative AI applications tailored to their ERP workloads. The program brings together enterprise technology from SAP and generative AI services from AWS with professional expertise from both parties — including teams of AI experts, professional services consultants, and solutions architects — to help support customers in their implementation journeys.

The program will include dedicated technical resources, cloud credits, and more to support the development, testing, and deployment of industry-specific applications.

“AWS and SAP’s long-standing partnership has helped customers accelerate their cloud journey and unlock more value from their business data,” said Ruba Borno, vice president of Specialists and Partners at AWS. “Our AI Co-Innovation Program is a significant next step that will give organizations the security and flexibility to build generative AI applications with Amazon Bedrock that can analyze and act on their most critical SAP data. This will help customers transform decades of business information into actionable insights while accelerating their path to becoming more agile, data-driven organizations.”

“Through the AI Co-Innovation Program with AWS, we’re enabling businesses to solve their most complex operational challenges with precision and speed,” said Philipp Herzig, CTO and chief AI officer at SAP. “By combining the power of our fully integrated platform with SAP BTP and our deep business process expertise with AWS’s comprehensive generative AI capabilities, partners can now create purpose-built AI agents that solve their most pressing challenges — identifying financial anomalies in real time to automatically optimizing supply chains during disruptions.”

The program also allows partners to rapidly build and scale generative AI applications using the latest generative AI tools and services from Amazon Bedrock, including large language models (LLMs) such as Amazon Nova and Anthropic Claude in AI Foundation on SAP Business Technology Platform (SAP BTP).

This announcement expands on the work AWS and SAP are doing to help customers — including Hyundai Motor Group, Moderna, and Zurich Insurance Group — modernize and move SAP workloads to AWS, realizing the availability, flexibility, and scalability of the cloud. Running SAP workloads on AWS allows customers to then combine their data with generative AI solutions. Partners including Accenture and Deloitte are among the first to work with AWS and SAP through the program, helping them accelerate the development and deployment of generative AI solutions to solve complex challenges.

“The AWS and SAP AI Co-Innovation Program brings together AWS cloud infrastructure and SAP enterprise software experience. Combined with Accenture’s AI transformation expertise and industry knowledge, we can show companies exactly how to integrate generative AI services with their most critical business workloads,” said Caspar Borggreve, senior managing director and SAP Business Group lead at Accenture. “For example, together with AWS and SAP, we are working with a utilities client to build a natural disaster asset resiliency capability to anticipate and respond to environmental challenges, protecting asset-intensive landscapes and maintaining service continuity for its customers.”

“This AI Co-Innovation Program combines cutting-edge generative AI capabilities from AWS and SAP with Deloitte’s deep industry experience and technology capabilities to deliver transformative solutions for our customers,” said Nishita Henry, AWS global chief commercial officer at Deloitte Consulting LLP. “Through the program, we are building a finance solution powered by Amazon Bedrock to help healthcare and life sciences companies optimize their product mix, improve forecast accuracy, and maintain competitive pricing, even during uncertain market conditions.”

For more details on the AWS SAP AI Co-Innovation Program, visit aws.amazon.com/sap/ai.

Kai Muehlbauer is head of AI Product and Partner Management at SAP.
Sara Alligood is global AWS head of SAP at Amazon Web Services.