Backend Observability: Requests, Queries, Queues & Dependencies

Backend Observability: Requests, Queries, Queues, and Dependencies

A backend can return 200 OK and still be unhealthy. Requests may be slowing down, database calls may be multiplying, jobs may be waiting in a queue, or an external API may be consuming most of the response time. CPU and memory graphs alone rarely explain these failures.

Backend observability means collecting enough signals to understand what a backend is doing, where time is being spent, what is failing, and why. For most application backends, four areas provide a useful starting model: requests, database queries, queues, and dependencies.

Direct Answer

Backend observability should let you follow work through the system:

request → application code → database → queue → worker → downstream dependency

Metrics show patterns such as rising latency or error rates. Traces connect operations across components. Logs provide detailed events and context. OpenTelemetry supports correlation between these signals through trace and span identifiers, while W3C Trace Context defines a standard mechanism for propagating tracing information between services. (opentelemetry.io)

The goal is not to collect everything. It is to collect enough evidence that an engineer can move from “the API is slow” to “this endpoint is waiting on this query, queue, or dependency.”

What Should You Observe?

This model complements broader service monitoring. Google’s SRE guidance describes latency, traffic, errors, and saturation as four fundamental monitoring signals for user-facing systems. (sre.google)

Start With Requests

Requests are usually the best entry point because they represent work that users or other systems actually asked the backend to perform.

For an HTTP API, useful measurements include:

• requests per second;
• response status or application outcome;
• request duration;
• latency distribution such as p50, p95, or p99;
• route or operation;
• service and deployment version.

Do not rely only on average response time. A service can have a reasonable average while a smaller percentage of requests are extremely slow. Google SRE specifically warns about latency tails, and histogram-style measurements make those distributions visible. Prometheus also documents histograms as a mechanism for representing distributions such as request duration. (sre.google)

OpenTelemetry defines standard semantic conventions for HTTP client and server spans. Using consistent attributes and operation names makes traces easier to compare across frameworks and services. (opentelemetry.io)

INTERNAL LINK: HTTP Request Performance Monitoring

Follow the Queries

Suppose /api/products becomes slow.

The request itself tells you that a problem exists, but not necessarily why. A trace might show something like:

GET /api/products             680 ms
├── authentication             12 ms
├── SELECT products           610 ms
└── response serialization     18 ms

Now the investigation has changed. Instead of optimizing the controller, you investigate the database operation.

OpenTelemetry defines database client spans for representing database operations observed by the calling application. (opentelemetry.io)

Useful database signals include query duration, call frequency, errors, connection-pool pressure, and a normalized query identity. Recording every literal SQL statement is not always appropriate: query text may contain sensitive or high-cardinality values, so instrumentation should follow the sanitization and data-handling rules of the database and observability stack.

For PostgreSQL, pg_stat_statements can aggregate planning and execution statistics for SQL statements, making it useful for identifying expensive or frequently executed query patterns. (postgresql.org)

Once a suspicious query is identified, EXPLAIN shows the execution plan. EXPLAIN ANALYZE actually executes the query and reports real row counts and execution timing for plan nodes, which can help verify whether estimates, scans, joins, or other operations are responsible for the observed behavior. (postgresql.org)

INTERNAL LINK: Diagnosing Slow SQL Queries

Observe Queue Time Separately

Queues change the meaning of latency.

Consider an image-processing job:

created
   ↓
waiting in queue: 14.2 s
   ↓
worker starts
   ↓
processing: 820 ms
   ↓
completed

Looking only at worker execution would suggest an 820 ms operation. From the application's perspective, however, the job took more than 15 seconds to complete.

That distinction matters.

Track at least the amount of waiting work, time spent waiting, processing duration, and failure or retry behavior where the queue technology exposes those signals. Queue depth alone can be misleading: ten lightweight jobs and ten expensive jobs do not necessarily represent the same workload.

Google SRE describes excessive queue length as both a latency and resource problem: queued requests consume resources and can amplify overload when work arrives faster than the system can process it. (sre.google)

Distributed tracing also becomes harder around asynchronous systems because the producer and consumer execute at different times. OpenTelemetry’s messaging conventions explicitly address propagation between producers and consumers so that operations can still be correlated across messaging infrastructure. (opentelemetry.io)

INTERNAL LINK: Queue Monitoring and Worker Performance

Measure Dependencies

Most production backends do not work alone. A request might depend on:

Application
├── PostgreSQL
├── Redis
├── payment API
├── object storage
└── internal user service

A dependency should therefore be observable as a distinct operation rather than disappearing inside total request duration.

For each important dependency, useful signals usually include latency, request volume, error rate, timeout behavior, and the calling operation. This allows an engineer to distinguish two very different situations:

API slow
└── application code slow

from:

API slow
└── payment provider: 2.4 s

That difference determines where the investigation should continue.

Connect the Entire Trace

Distributed observability becomes much more useful when the same operation can be followed across service boundaries.

W3C Trace Context defines the traceparent and tracestate HTTP headers for this purpose. traceparent carries standardized trace information, allowing cooperating systems to preserve the relationship between operations even when different tracing implementations are involved. (w3.org)

Conceptually:

Browser
   │
   ▼
API Gateway
trace-id: abc
   │
   ▼
Order Service
trace-id: abc
   │
   ├── PostgreSQL
   │     trace-id: abc
   │
   └── Payment Service
         trace-id: abc

OpenTelemetry calls this context propagation. A downstream service receives the trace context and creates its own span as part of the same distributed trace. (opentelemetry.io)

Without propagation, engineers see isolated operations. With propagation, they see a causal path.

A Practical Diagnostic Flow

Imagine customers report that checkout occasionally takes five seconds.

Start with the request metric. Confirm whether checkout latency actually increased and whether the problem affects all requests or primarily the slow tail.

Then open a slow trace:

POST /checkout               5.1 s
├── validate cart             8 ms
├── SELECT inventory         31 ms
├── payment request           4.8 s
└── create order             76 ms

The database is probably not the first place to investigate. The payment dependency dominates the trace.

Check dependency metrics and logs for the same period. Look for increased remote latency, timeouts, retries, or connection failures. After making a change, measure the same request and dependency signals again.

The workflow is:

symptom → measurement → suspected component → confirmation → fix → re-measurement

That is much more reliable than optimizing whichever component happens to look suspicious first.

Common Observability Mistakes

The first mistake is collecting infrastructure metrics without application context. CPU usage can reveal saturation, but it cannot tell you which endpoint or query caused it.

The second is logging everything while correlating nothing. A million log records are difficult to investigate if they cannot be connected to the request or job that generated them. OpenTelemetry supports attaching trace and span identifiers to log records specifically to enable this correlation. (opentelemetry.io)

The third is ignoring asynchronous waiting time. Worker processing time and end-to-end job latency are different measurements.

The fourth is using averages as the entire latency story. Tail latency may reveal problems hidden by an acceptable mean.

Finally, avoid uncontrolled telemetry cardinality. User IDs, full URLs with arbitrary parameters, raw SQL values, request bodies, or unique job IDs can make metric systems expensive and may expose sensitive data. Keep high-cardinality diagnostic context in systems designed to handle it rather than blindly turning every value into a metric label.

SeoNest Recommendation

Start from the request boundary and instrument outward.

For each critical backend operation, establish request latency, traffic, and errors first. Then expose database calls, messaging operations, and important downstream dependencies as traceable operations. Preserve trace context across HTTP calls and asynchronous messages, and correlate logs with the same execution context.

Do not begin by building dozens of dashboards. Begin with a smaller question:

Can an engineer take one slow or failed request and determine where the time or failure came from?

If the answer is no, the backend is still missing important observability.

FAQ

Are logs enough?

Usually not. Logs provide detailed events, but metrics are better for detecting aggregate patterns, while traces show how operations relate across a request. Correlating all three is more useful than treating them as isolated datasets.

Should every SQL query become a trace span?

Useful database operations should be observable, but instrumentation volume, sensitive query data, and telemetry cost matter. Automatic instrumentation should be configured deliberately rather than assumed to be safe at unlimited volume.

What should I monitor for queues?

Queue or backlog size, time spent waiting, processing duration, failures, retries, and worker capacity are useful starting points. The exact signals depend on the messaging system and workload.

What is the difference between monitoring and observability?

Monitoring commonly tracks predefined system signals and conditions. Observability uses telemetry such as metrics, traces, and logs to investigate system behavior, including failures that were not predicted when dashboards or alerts were created.

Does distributed tracing require OpenTelemetry?

No. W3C Trace Context is vendor-neutral, and different tracing implementations can support distributed context propagation. OpenTelemetry provides widely adopted APIs, SDKs, semantic conventions, and tooling for producing and transporting telemetry. (w3.org)

Final Takeaway

Good backend observability turns latency from a vague symptom into a sequence of measurable operations.

Do not stop at “the request took two seconds.”

Determine whether those two seconds were spent executing application code, waiting for SQL, sitting in a queue, or calling another service. Requests reveal the symptom; queries, queues, and dependencies reveal where the work went; traces and correlated logs connect the evidence.

That is the difference between knowing that a backend is unhealthy and being able to explain why.

Sources

1. W3C — Trace Context. W3C Recommendation. Defines standardized traceparent and tracestate propagation. W3C Trace Context
2. OpenTelemetry — Semantic Conventions for HTTP Spans. Official OpenTelemetry specification. OpenTelemetry HTTP Spans
3. OpenTelemetry — Semantic Conventions for Database Client Spans. Official OpenTelemetry specification. OpenTelemetry Database Spans
4. OpenTelemetry — Semantic Conventions for Messaging Spans. Official OpenTelemetry specification. OpenTelemetry Messaging Spans
5. OpenTelemetry — Context Propagation and Logging Specification. Official documentation covering distributed context and log correlation. OpenTelemetry Context Propagation OpenTelemetry Logging
6. Google — Site Reliability Engineering: Monitoring Distributed Systems. Rob Ewaschuk, edited by Betsy Beyer. Covers latency, traffic, errors, saturation, and tail latency. Google SRE — Monitoring Distributed Systems
7. Google — Site Reliability Engineering: Addressing Cascading Failures. Covers queueing, overload, latency, and saturation behavior. Google SRE — Addressing Cascading Failures
8. Prometheus — Histograms and Summaries. Official documentation on representing distributions such as request latency. Prometheus Histograms and Summaries
9. PostgreSQL — pg_stat_statements and Using EXPLAIN. Official PostgreSQL documentation for query statistics and execution-plan analysis. PostgreSQL pg_stat_statements PostgreSQL Using EXPLAIN