17.2.3.3 Shutdown Connection Closing

Shutdown connection closing addresses the specific mechanics of correctly closing each individual downstream connection type, a database pool, a cache client, a message queue consumer, an HTTP keep-alive client to another service, during the shutdown sequence, since each connection type has its own correct closing procedure and its own risk of leaking resources or leaving the remote side in an inconsistent state if closed incorrectly.

Database connection pools

A connection pool should be closed through its own dedicated method, which waits for any connections currently checked out and in use to be returned before actually closing them, rather than abruptly terminating connections that might still be mid-query:

await pool.end();

pool.removeAllListeners();
process.exit(0); // does not actually close database connections

Most database client libraries provide an end() or equivalent method specifically designed for this graceful pool closure; bypassing it and simply exiting the process can leave the database server with connections it believes are still active, which can cause confusion or resource exhaustion on the database side until those abandoned connections eventually time out independently.

Message queue consumers

A message queue consumer should stop pulling new messages first, finish processing whatever message it is currently handling, and only then close its actual connection to the broker, since abruptly disconnecting while a message is checked out but not yet acknowledged typically causes that message to be redelivered to a different consumer, which is wasteful but generally not incorrect, as opposed to a message being lost entirely if acknowledgment handling itself is interrupted mid-way:

let shuttingDown = false;
channel.consume(queue, async (msg) => {
  if (shuttingDown) return channel.nack(msg, false, true); // requeue rather than process during shutdown
  await processMessage(msg);
  channel.ack(msg);
});

process.on('SIGTERM', async () => {
  shuttingDown = true;
  await currentMessageProcessing;
  await channel.close();
  await connection.close();
});

Closing the channel before the connection, in that specific order, follows the underlying protocol's own expected closure sequence for most message broker client libraries.

Cache client connections

Cache clients are often more tolerant of an abrupt closure than a database, since cached data is, by definition, not the sole source of truth and a lost cache connection generally just means a cache miss on the next request rather than data loss, but closing cleanly still avoids leaving the cache server tracking a connection that has actually gone away:

await redisClient.quit();

redisClient.disconnect(); // abrupt, less graceful than quit()

Using the client library's graceful quit() method where available, rather than an abrupt disconnect, sends a proper closing handshake to the cache server rather than simply dropping the underlying socket.

Outbound HTTP clients with keep-alive connections

An HTTP client maintaining keep-alive connections to other internal services should release those connections cleanly, since an abruptly terminated connection can leave the remote service's own connection pool holding a now-dead socket until it independently times out:

httpAgent.destroy();

For most HTTP client libraries, this is a relatively low-risk closure compared to a database or message queue, since HTTP connections are generally designed to handle this kind of termination gracefully on the receiving end, but explicitly destroying the agent's connection pool during shutdown remains good practice rather than simply allowing the process exit to implicitly close every open socket without any explicit cleanup step.

Ordering closure across multiple, interdependent connection types

When a shutdown sequence needs to close several different connection types, the order matters specifically where one type's closure might still be needed to complete in-flight work that another step depends on; generally, closing connections to systems no longer needed first, then proceeding to ones still required for completing final, in-flight work, then finally closing those as the very last step:

process.on('SIGTERM', async () => {
  server.close(); // stop accepting new HTTP requests
  await waitForInFlightRequests(); // these may still use the db pool and cache client
  await redisClient.quit();
  await pool.end();
  process.exit(0);
});

Closing the database pool or cache client before in-flight requests that still depend on them have actually finished would cause those in-flight requests to fail with a connection error, precisely the outcome graceful shutdown is meant to avoid.

Handling a connection that fails to close within a reasonable time

A specific connection's close operation that hangs indefinitely, due to the remote server being unresponsive or a bug in the client library itself, should not be allowed to block the entire shutdown sequence indefinitely; wrapping each closure in its own timeout ensures the overall shutdown sequence still converges even if one specific connection's closure does not complete cleanly:

await Promise.race([
  pool.end(),
  new Promise((resolve) => setTimeout(resolve, 5000)),
]);

This ensures the shutdown sequence proceeds to its next step, and ultimately to process exit, even if this specific connection closure did not complete within a reasonable window, trading a theoretically perfect cleanup for the practical certainty that the overall shutdown process will actually complete within the container's available stop timeout.

Verifying no connections leak across repeated restarts

Periodically confirming that repeated container restarts do not produce an accumulating number of abandoned, never-properly-closed connections on a shared downstream resource, such as a database, validates that the shutdown sequence is genuinely closing connections correctly rather than merely appearing to:

SELECT count(*) FROM pg_stat_activity WHERE application_name = 'my-api';

A connection count that grows unboundedly across many restarts, rather than remaining roughly stable, is a direct, measurable signal that connections are not actually being closed correctly during shutdown, regardless of how the shutdown handler's code appears to be structured.

Common mistakes

• Exiting the process without explicitly closing a database connection pool through its dedicated, graceful closure method, leaving the database server believing connections are still active.
• Closing a message queue connection before unacknowledged, in-progress messages have been handled, risking unnecessary redelivery or, in the worst case, message loss.
• Closing downstream connections before in-flight requests that still depend on them have actually finished using them.
• Not wrapping individual connection closures in their own timeout, allowing one hung closure to block the entire shutdown sequence indefinitely.
• Never verifying connection counts on a shared downstream resource across repeated restarts, missing a gradual, accumulating connection leak that the shutdown sequence's code appears to prevent but does not actually close successfully in practice.

Shutdown connection closing requires treating each downstream connection type according to its own correct closure procedure, ordering closures to respect what in-flight work still depends on, bounding each closure with its own timeout, and periodically verifying through actual connection counts that the shutdown sequence is genuinely closing connections rather than merely appearing to in code review.