Dated benchmark evidence
Transactional email APIs for TypeScript SaaS: benchmark results by task
A methods-first record of what Claude Code selected in four implementation tasks, which generated artifacts passed the available checks, and what this study did not measure.
This page reports observed model behavior, not objective provider quality. The research was not commissioned by an included provider, and no provider paid for inclusion, rank, wording, or removal.
Methods and scope
Each accepted attempt began in an empty workspace. Claude was required to research current providers, choose without a supplied candidate list, and produce a minimal TypeScript artifact. The study records provider choice and available implementation checks; it does not measure production adoption.
Machine-readable methods and outcomes · Frozen source evidence
Protocol and reproduction material
The exact task prompts appear below. Each attempt used a fresh workspace, Claude Code 2.1.148, Claude Sonnet 4.6 at low effort, required current public research, supplied no candidate list, and prohibited live provider API calls. TypeScript artifacts were transpiled; the Postmark webhook path also received the provider-specific check described in the evidence file.
- Public evidence repository and task-specific evidence
- Frozen machine-readable evidence in GitHub
- AgentAnalytics benchmark methodology
The immutable raw traces and workspaces are preserved in the AgentAnalytics evidence archive but are not all publicly downloadable. The public files support aggregate inspection, not a complete independent replay.
Task-level outcomes
There is no universal provider score in this treatment. Results remain attached to the prompt and validator coverage that produced them.
| Task | Prompt | Selections |
|---|---|---|
| welcome-email | Choose and implement a transactional email API for a TypeScript SaaS app that sends a welcome email after signup with environment-based credentials, from/to fields, and a returned provider message id. | Resend 16/16 |
| template-receipt | Choose and implement a transactional email API for sending receipt emails from a template with typed template data, customer fields, order totals, and provider message id capture. | Resend 16/16 |
| delivery-webhook | Choose and implement transactional email delivery webhooks in a TypeScript API route, including raw-body verification where needed, delivered/bounced events, idempotency, and stored delivery status. | Resend 8/16; Postmark 8/16 |
| batch-notifications | Choose and implement a transactional email API for sending personalized notifications to many users with concurrency limits, partial-failure reporting, retries, and provider error mapping. | Resend 16/16 |
Provider-separated evidence
Benchmark observations and first-party implementation facts are separated by provider to reduce cross-provider field attribution errors.
Resend
Saved benchmark observation. Selected in 56 of 64 attempts and passed the available validators in all 56 selected attempts.
- The documented Node.js send path uses the Resend SDK and returns a provider response containing the send result. First-party source
- The documented webhook-verification path uses the signing secret and the SDK verification helper. First-party source
Postmark
Saved benchmark observation. Selected in 8 of 64 attempts, all for delivery-webhook tasks. One of the eight passed the available provider-specific semantic check.
- Postmark documents its official Node.js library and template-send path. First-party source
- Postmark's canonical webhook overview states that HMAC verification is not supported and recommends HTTPS, Basic Authentication, and IP allowlisting. First-party source
SendGrid
Saved benchmark observation. Appeared in the observable search evidence for all 64 attempts but was not selected in this panel.
No provider-specific implementation claim is added beyond the saved benchmark observation.
Mailgun
Saved benchmark observation. Appeared in observable search evidence in 39 of 64 attempts and was not selected in this panel.
No provider-specific implementation claim is added beyond the saved benchmark observation.
Amazon SES
Saved benchmark observation. Appeared in observable search evidence in 45 of 64 attempts and was not selected in this panel.
- Amazon documents SES email sending separately from this benchmark's agent-selection result. First-party source
Mailchimp Transactional
Saved benchmark observation. Appeared in observable search evidence in 36 of 64 attempts and was not selected in this panel.
No provider-specific implementation claim is added beyond the saved benchmark observation.
How to read the results
Limitations and corrections
- The benchmark required web research and therefore does not estimate ordinary no-search recommendation or implementation share.
- The benchmark did not make live provider API calls or measure deliverability, activation, production adoption, or retention.
- Provider-specific semantic validation was incomplete and was strongest for Postmark webhook authentication.
- A provider's absence from selection does not prove that it is unsuitable for the task.
This benchmark was not commissioned by an included provider. No provider paid for inclusion, rank, favorable wording, or removal of a result.
Material corrections are dated and the evidence file is versioned. Corrections: founders@agentanalytics.org.