May 18, 2026 · 6 min read
Designing conversational backends around API Gateway timeouts
The 29-second default shaped a generation of bot architectures. What changed, what didn't, and the patterns that keep slow core systems from freezing your conversations.
Every conversational architect learns the number 29 eventually. API Gateway's classic 29-second integration timeout — and the tighter patience of voice channels — means a slow policy-admin system can stall a conversation past the point a customer will wait. AWS has since made the integration timeout configurable beyond 29 seconds for Regional REST APIs (with throttle trade-offs), but the design lesson stands: a conversation should never synchronously wait on a slow core system.
Pattern one: acknowledge, then callback
For genuinely slow operations — endorsements, document generation — return immediately with a conversational acknowledgement, hand the work to an asynchronous path (SQS or Step Functions), and notify on completion: an outbound message, or a status the bot checks when the customer asks. Customers accept “I’ve started this; you’ll get an SMS within a few minutes” far better than dead air.
Pattern two: cache the conversation's working set
Most turns don't need a live core-system read. A short-TTL DynamoDB cache of the session's working set — the policy summary fetched once at authentication — turns five slow calls into one, and makes every subsequent turn feel instant.
Pattern three: budget every hop
Set explicit timeout budgets per hop (bot → orchestrator → adapter → core) that sum comfortably under the channel's patience, with circuit breakers that degrade gracefully to “let me connect you to an agent” instead of a stack trace. In voice, aim for total turn latency under three seconds; fill unavoidable waits with progress speech.
None of this is exotic. It's the difference between a bot that demos well and one that survives contact with a mainframe.