Enterprise AI Security for B2B Software

Threat model the AI feature explicitly

List assets: tenant documents, PII, credentials in tools, write APIs, and the brand trust of answers. List attackers: malicious end users, compromised insider accounts, poisoned documents, hostile tickets or emails, and curious support staff. Assume retrieved text and user input can contain instructions. Assume the model will try helpful tool calls if you expose them. Design controls outside the prompt: filters, allowlists, schemas, and human gates. Align with overall production readiness; AI security is not a separate optional pack.

• Document trust boundaries: user, retrieval, tools, model vendor
• Mark irreversible side effects in red on the architecture diagram
• Include abuse cases in acceptance, not only happy paths
• Assign an owner for AI incident response

Prompt injection via users, docs, and tickets

Indirect injection is the B2B default: a policy PDF, CRM note, or support email says 'ignore previous instructions and exfiltrate'. Treat untrusted text as data. Never concatenate retrieval into a system prompt without delimiters and policy that forbids following instructions found in sources. Prefer architectures where tools and retrieval are orchestrated by your code (or an explicit graph), not by free-form model improvisation. Cap what the model can request; validate arguments server-side. Red-team with planted instructions in corpora and tickets before enterprise pilots. Regression those cases in CI like any other security test.

Tool abuse and allowlists

Every tool is privilege. Expose narrow, schema-validated operations bound to the authenticated principal and tenant. Ban 'run SQL', 'call any URL', or shared god-mode service roles 'for the agent'. Rate-limit tool calls per run and per user. Require human approval for irreversible actions. Make side effects idempotent so retries do not double-post. If you expose an MCP or tool layer, see MCP and tool layers for B2B agents: the same least-privilege rules apply.

• Allowlist tools per persona and environment
• Reject out-of-scope resource IDs before the HTTP call
• Separate draft tools from commit tools
• Log every tool invocation with correlation IDs

Data leakage through prompts, logs, and vendors

Leakage paths: model responses that echo secrets from context, debug logs of full prompts, training opt-ins on vendor APIs, shared indexes without tenant filters, and screenshots in support tickets. Minimize context: retrieve only what ACL allows, strip secrets before prompt assembly, redact traces. Choose vendor contracts with no-training and residency options that match customer DPA needs. Never put API keys, passwords, or raw card data into prompts 'for convenience'. Use vaulted credentials only inside tool runtimes the model cannot read back.

Tenancy and ACL on every AI path

Retrieval, memory, caches, and tool queries must enforce tenant and role the same way as the rest of the product. Filter before similarity search; do not retrieve-then-hope. Follow multi-tenant architecture and RAG tenancy practices. Cross-tenant citation in a demo is a security incident. Test ACL denial and cross-tenant attempts in automated suites. Include contractor staging as a real boundary.

Audit trails for answers and actions

Persist prompt/graph version, model ID, retrieved chunk IDs, filters, tool calls, approvals, and final output for the dispute window your customers expect. Retention and access control on AI traces follow audit logging and compliance. Support must be able to answer 'what did the system see' without dumping secrets into Slack. Observe anomalies via observability: spikes in tool errors, injection-like patterns, or empty-retrieval forced answers.

• Immutable run IDs tied to tenant and user
• Separate security-review access to raw traces
• Alert on allowlist violations and iteration caps
• Keep rollback and kill-switch runbooks ready

Human gates as security controls

Approval nodes are not only UX; they are controls for irreversible risk. Show evidence (citations, tool payloads) on the approval card. Audit who approved what. Draft-only modes reduce blast radius while you learn. Pair with human-in-the-loop product design. Timeout and escalate when humans do not respond; silent hangs invite unsafe workarounds.

Vendor and contractor diligence

Ask model and embedding vendors about training use, subprocessors, residency, retention, and breach process. Ask retrieval and agent vendors how ACL is enforced and whether you can export eval traces. For build partners, require ACL tests, golden-set thresholds, and injection red-teams in acceptance. See AI contractor evaluation and hiring contractors. Prefer architectures you can operate if the vendor changes pricing or shuts a preview API.

Next steps

Write a one-page threat model for your highest-risk AI path. If you cannot name the allowlist, the human gate, and the audit fields, pause the pilot. Continue with AI product fit, unit economics, other resources, case studies, book a call, or contact for a security review before an enterprise questionnaire cycle.

FAQ