During a hot restart, the parent forwards UDP packets it can no longer dispatch (new QUIC connections and any session it does not own after drainListeners) to the child over the _udp unix datagram socket. It does so from its main thread: HotRestartingParent::sendHotRestartMessage posts to the dispatcher, and RpcStream::sendHotRestartMessage puts the socket into blocking mode and calls sendmsg() with no timeout (hot_restarting_base.cc). The child's inherited UDP listeners stay paused until the parent terminates (udp_listener_impl.cc), so every new QUIC packet takes this path for the whole --parent-shutdown-time-s window.
The child's main thread meanwhile runs its stats flush, which calls mergeParentStatsIfAny → HotRestartingChild::getParentStats → receiveHotRestartMessage(Blocking::Yes): a blocking recvmsg() on the main socket with no timeout, during which nothing reads the _udp socket. duplicateParentListenSocket (a listener added during the window) has the same shape.
Once the child's _udp socket queue is full (a few dozen forwarded datagrams), the parent's main thread is parked in sendmsg() and never reads the child's stats request; the child's main thread is parked in recvmsg() and never drains the _udp socket. Both wait on each other for good. Workers keep serving existing connections, but admin, xDS, timers and signal handling are dead in both processes and the parent never gets its terminate request (SIGTERM cannot land either).
Repro / evidence
Reproduced on a deployment with a few hundred HTTP/3 requests per second across many UDP listeners, by briefly freezing the child's main thread during the drain window so the forwarding socket fills. Stacks on the wedged pair:
- parent:
sendmsg ← RpcStream::sendHotRestartMessage ← DispatcherImpl::runPostCallbacks (fd is the _udp parent socket)
- child:
recvmsg ← RpcStream::receiveHotRestartMessage ← HotRestartingChild::getParentStats ← HotRestartImpl::mergeParentStatsIfAny ← InstanceBase::updateServerStats ← flushStatsInternal
--skip-hot-restart-parent-stats avoids the most frequent trigger (the stats exchange) but not the listen socket hand-off one.
This path has existed since UDP forwarding was added in #29585; it takes enough QUIC traffic through the drain window to fill the socket.
Related
RpcStream::sendHotRestartMessage also sleeps 10×1 s and then RELEASE_ASSERTs on ECONNREFUSED, so a child that dies during the window stalls and then crashes the parent, and a parent that dies makes the child's next request abort.
I have a fix (parent forwards through a bounded non-blocking queue; the child keeps servicing forwarded packets while it waits for a reply, with a bounded wait; the sockets get send/receive timeouts) and will open a PR.
During a hot restart, the parent forwards UDP packets it can no longer dispatch (new QUIC connections and any session it does not own after
drainListeners) to the child over the_udpunix datagram socket. It does so from its main thread:HotRestartingParent::sendHotRestartMessageposts to the dispatcher, andRpcStream::sendHotRestartMessageputs the socket into blocking mode and callssendmsg()with no timeout (hot_restarting_base.cc). The child's inherited UDP listeners stay paused until the parent terminates (udp_listener_impl.cc), so every new QUIC packet takes this path for the whole--parent-shutdown-time-swindow.The child's main thread meanwhile runs its stats flush, which calls
mergeParentStatsIfAny→HotRestartingChild::getParentStats→receiveHotRestartMessage(Blocking::Yes): a blockingrecvmsg()on the main socket with no timeout, during which nothing reads the_udpsocket.duplicateParentListenSocket(a listener added during the window) has the same shape.Once the child's
_udpsocket queue is full (a few dozen forwarded datagrams), the parent's main thread is parked insendmsg()and never reads the child's stats request; the child's main thread is parked inrecvmsg()and never drains the_udpsocket. Both wait on each other for good. Workers keep serving existing connections, but admin, xDS, timers and signal handling are dead in both processes and the parent never gets its terminate request (SIGTERM cannot land either).Repro / evidence
Reproduced on a deployment with a few hundred HTTP/3 requests per second across many UDP listeners, by briefly freezing the child's main thread during the drain window so the forwarding socket fills. Stacks on the wedged pair:
sendmsg←RpcStream::sendHotRestartMessage←DispatcherImpl::runPostCallbacks(fd is the_udpparent socket)recvmsg←RpcStream::receiveHotRestartMessage←HotRestartingChild::getParentStats←HotRestartImpl::mergeParentStatsIfAny←InstanceBase::updateServerStats←flushStatsInternal--skip-hot-restart-parent-statsavoids the most frequent trigger (the stats exchange) but not the listen socket hand-off one.This path has existed since UDP forwarding was added in #29585; it takes enough QUIC traffic through the drain window to fill the socket.
Related
RpcStream::sendHotRestartMessagealso sleeps 10×1 s and thenRELEASE_ASSERTs onECONNREFUSED, so a child that dies during the window stalls and then crashes the parent, and a parent that dies makes the child's next request abort.I have a fix (parent forwards through a bounded non-blocking queue; the child keeps servicing forwarded packets while it waits for a reply, with a bounded wait; the sockets get send/receive timeouts) and will open a PR.