Case study 01 · Katalon

From natural language to safe actions

An agentic assistant embedded in a testing platform: users need grounded actions, not a disconnected general-purpose chatbot.

Context

Testers ask for test cases, create test assets, execute suites and investigate results from inside the product. A single request crosses both AI and traditional software boundaries, so safe execution needs explicit identity, validation and observable tool calls.

Containers

CopilotKit UI
Embedded chat, page context, client-side UI tools.
AG-UI client
Runtime / BFF
Identity, history, AG-UI ↔ A2A bridge, task management.
Node.js
Kai orchestrator
Google ADK: tool use, skills, reasoning and delegation.
Python agent
Protected MCP
Domain tool schemas, authorization and dispatch.
FastMCP
Backend APIs
TestOps microservices and TestCloud execution.
Source of truth
Nango connector
OAuth lifecycle and provider API proxy.
External auth

Calls

CopilotKit UI —AG-UI→ Runtime / BFF
Runtime / BFF —A2A→ Kai orchestrator
Kai orchestrator —MCP→ Protected MCP
Protected MCP —HTTP→ Backend APIs
Protected MCP —scoped proxy→ Nango connector

Principles

Fail-closed tool discovery

If a connection is absent, disabled or unavailable, the exposed tool set is restricted — and checked again at invocation time.

Multi-tenant credential mapping

External connections resolve from the authenticated user and account. No cross-tenant lookup, no model-selected credentials.

Separate reads and writes

A bounded write-tool allowlist limits mutations; higher-impact actions require explicit approval controls.

Tokens never reach the LLM

Nango brokers OAuth; the MCP layer passes a scoped proxy, so the model only ever sees results.

Decisions

ADR-01

Agent vs. deterministic code

Let models interpret intent and synthesize language. Gate permissions, calculations and writes in code.

TRADE-OFF Auditability over maximal model autonomy.

ADR-03

Agent delegation vs. tool calling

Delegate specialist reasoning with A2A child tasks; use MCP for bounded backend capabilities. TestCloud execution stays a domain tool call.

TRADE-OFF Clear task boundaries and safe execution.

ADR-04

In-memory cache vs. live state

A connector status cached per pod went stale on other replicas. For low-frequency, user-mutable state, read the source of truth.

TRADE-OFF Consistency over minor read savings.

My scope

Google ADK agent harness and platform integration
FastMCP tools and Nango-backed OAuth connectors
Natural-language execution integration with TestCloud
Full-stack and distributed execution platform experience

Architecture boxes describe component responsibilities, not proof of production rollout. No customer counts, benchmark results or SLO attainment are claimed.