15.1.1.1 App Log Output

App log output is the content an application itself decides to write, as distinct from the transport mechanism (stdout, a logging driver, an aggregation pipeline) that carries it, and the quality of that content, what is logged, in what format, at what level of detail, determines how useful the entire observability pipeline built around it actually turns out to be.

Designing what belongs in a log line

A useful log line answers a specific question an operator might ask later: what happened, when, to which entity, and with what outcome. Logging too little leaves gaps during an investigation; logging too much buries the signal in noise and increases storage and query cost without a corresponding benefit:

logger.info('Order processed', {
  orderId: order.id,
  userId: order.userId,
  amount: order.total,
  durationMs: Date.now() - startTime,
});

This line answers what happened (an order was processed), to which entity (a specific order and user), and includes a measurable outcome (processing duration), which together make it useful both for debugging a specific incident and for aggregating across many log lines to understand typical behavior.

Choosing the right log level

Most logging libraries support a hierarchy of severity levels, and using them consistently is what makes filtering by severity during an investigation actually effective:

logger.debug('Cache lookup', { key, hit: false });
logger.info('User logged in', { userId });
logger.warn('Retrying database query', { attempt: 2 });
logger.error('Payment processing failed', { orderId, error: err.message });

A common failure mode is logging routine, expected events at error level out of habit or because the developer wanted them to be visible, which trains operators to ignore error-level alerts because they are too frequently triggered by non-incidents, defeating the purpose of having a severity hierarchy at all.

Including correlation identifiers

A single user request often produces several log lines across the lifetime of handling it, and including a shared identifier across all of them is what allows those lines to be reconstructed as a coherent sequence later:

app.use((req, res, next) => {
  req.requestId = crypto.randomUUID();
  next();
});

logger.info('Request received', { requestId: req.requestId, path: req.path });
logger.info('Database query executed', { requestId: req.requestId, query: 'SELECT...' });
logger.info('Request completed', { requestId: req.requestId, status: res.statusCode });

In a multi-service architecture, this identifier should propagate across service boundaries (typically through a request header) so that a single trace ID can correlate log output produced by entirely different containers handling the same logical request.

const requestId = req.headers['x-request-id'] || crypto.randomUUID();
fetch(downstreamUrl, { headers: { 'x-request-id': requestId } });

Logging errors with enough context to act on

An error log line that only contains a generic message and a stack trace is often insufficient to actually diagnose the problem without additional context about what the application was trying to do at the time:

try {
  await chargePayment(order);
} catch (err) {
  logger.error('Payment charge failed', {
    orderId: order.id,
    amount: order.total,
    provider: 'stripe',
    errorMessage: err.message,
    errorCode: err.code,
  });
  throw err;
}

Including the relevant business context (which order, which amount, which provider) alongside the technical error details turns a log line from "something failed" into "this specific operation failed, here is exactly what was being attempted," which is the difference between a log that requires further investigation to even understand the scope of the problem and one that already answers most of the immediate questions.

Avoiding logging in tight loops

Logging inside a loop that runs many times per request, or many times per second, can produce an overwhelming volume of output that drowns out genuinely important log lines and adds meaningful overhead to the request path itself:

for (const item of items) {
  logger.debug('Processing item', { itemId: item.id }); // risky if items is large
}

logger.info('Processing batch', { itemCount: items.length });
// process items without per-item logging
logger.info('Batch processed', { itemCount: items.length, durationMs: elapsed });

Summarizing a batch operation with a single before-and-after log line, rather than one line per item, generally provides equivalent diagnostic value at a small fraction of the log volume.

Logging at application boundaries

The most consistently valuable log lines tend to sit at the boundaries of the application: incoming requests, outgoing calls to external services, and database queries, since these are the points where failure is most likely to originate and where timing information is most directly actionable:

const start = Date.now();
const response = await fetch(externalApiUrl);
logger.info('External API call', {
  url: externalApiUrl,
  status: response.status,
  durationMs: Date.now() - start,
});

Consistently logging at these boundaries, across every service in an architecture, builds a reliable, comparable picture of where time is actually spent and where failures actually originate, rather than relying on inconsistent, ad hoc logging that varies by which engineer happened to write a particular piece of code.

Common mistakes

• Logging routine, expected events at error level, training operators to tune out error alerts because they fire too frequently for non-incidents.
• Omitting business context from error logs, leaving a log line that confirms something failed without explaining what was actually being attempted.
• Logging inside tight loops at a volume that drowns out other signal and adds measurable overhead to the request path.
• Failing to propagate a correlation identifier across service boundaries, making it difficult to reconstruct a multi-service request's full path during an investigation.
• Writing inconsistent log formats and field names across different parts of the same application, undermining the value of structured logging once those logs are aggregated and queried together.

The quality of app log output, not the transport mechanism carrying it, ultimately determines whether an observability pipeline is genuinely useful during an investigation, and that quality comes from deliberate choices about what to log, at what level, with what context, and consistently enough across an application that the resulting logs can be trusted to answer the questions they will eventually be asked.