HTTP Response Shaping

1. What this document is about

This document covers response field selection (also known as sparse fieldsets) in REST APIs: a mechanism that allows clients to request only the fields they actually need, instead of receiving a fixed, full representation of a resource.

This document applies to:

• REST APIs with multiple consumers (frontend, mobile, integrations, patterns)
• APIs under high traffic or strict latency budgets
• Systems where payload size, CPU usage and serialization cost matter
• Architectures that need evolution without breaking consumers

This document does not apply to:

• Internal-only APIs with a single tightly-coupled client
• CRUD-style systems with trivial payloads and low traffic
• APIs where GraphQL is already the chosen contract model
• Systems where response shapping would violate regulatory or auditing requirements

---

2. Why this matters in real systems

In real systems, APIs rarely stay simple.

What starts as:

GET /customers/{id}

with a single frontend client, eventually becomes:

• Web frontend
• Mobile apps
• Background jobs
• External partners
• Internal services with different data needs

Typical pressures that surface:

• Payloads grow faster than clients actually need
• Serialization and deserialization become a measurable cost
• Small UI changes require new endpoints or DTOs
• Breaking changes accumulate due to rigid response contracts
• Over-fetching becomes the norm, not the exception

When field selection is ignored:

• APIs return wasted data on every request
• p95/p99 latency degrades silently as payloads grow
• Cache efficiency drops due to larger response bodies
• Teams create endpoint explosion (/customers/summary,/customers/details, /customers/mobile, etc.)

Simper approaches stop working because:

• DTO proliferation becomes unmanageable
• Versioning every representation is not scalable
• Clients evolve faster than backend release cycles

Field selection emerges as a response to evolution pressure, not as an optimization trick.

---

3. Core concept (mental model)

Think of a REST resource as a logical dataset not a fixed JSON shape.

The mental model is:

The server owns the data and invariants. The client declares which projection of that data it wants.

Conceptually:

1. The client sends a request with a field projection
2. The server validates that projection
3. The server executes a query optimized for that projection
4. The server returns only the selected fields

This is not dynamic typing. This is controlled projection.

A useful analogy is SQL SELECT lists:

SELECT Id, Name, Status FROM Customers

You are not changing what the table is. You are controlling which columns are materialized.

Field selection in REST applies the same idea to HTTP representations.

---

4. How it works (step-by-step)

Step 1 — Client expresses intent

The client explicitly declares required fields, usually via a query parameter:

GET /customers/42?fields=id,name,status

Assumption:

• The client knows what it needs
• The API documents supported fields

Invariant:

• Fields not requested must not be returned

---

Step 2 — API parses and validates the field set

The server parses the fields parameter into a strctured representation.

Validation includes:

• Unknown fields
• Forbidden fields (authorization)
• Conflicting or invalid combinations

Why this exists:

• Prevents leaking internal or sensitive fields
• Avoids runtime reflection chaos
• Keeps the API contract explicit

Invariant:

• Field selection is opt-in, never implicit

---

Step 3 — Projection is applied at the data access layer

The projection must influence data fetching, not just serialization.

This is critical.

Correct:

• Database query fetches only required columns

Incorrect:

• Load full entity → serialize selectively

Why:

• Avoids unnecessary IO
• Avoids EF Core tracking overhead
• Reduces memory allocations

Invariant:

• Over-fetching is considered a bug, not a convenience

---

Step 4 — Response is shaped deterministically

The response serializer outputs only requested fields.

Guarantees:

• Stable field naming
• No ordering guarantees unless documented
• Missing fields are omitted, not null

Invariant:

• Absence means "not requested", not "unknown"

---

5. Minimal but realistic example (.NET)

request

GET /customers/42?fields=id,name,status

---

Domain model (simplified)

Customer.cs

public sealed class Customer
{
    public Guid Id { get; init; }
    public string Name { get; init; } = null!;
    public string Email { get; init; } = null!;
    public CustomerStatus Status { get; init; }
    public DateTime CreatedAt { get; init; }
}

---

Field definition contract

Explicit, server-owned mapping

CustomerFields.cs

public static class CustomerFields
{
    public static readonly IReadOnlyDictionary<string, Expression<Func<Customer, object>>> Map =
        new Dictionary<string, Expression<Func<Customer, object>>>(StringComparer.OrdinalIgnoreCase)
        {
            ["id"] = c => c.Id,
            ["name"] = c => c.Name,
            ["email"] = c => c.Email,
            ["status"] = c => c.Status
        };
}

Why this matters:

• No reflection
• No stringly-typed access to domain objects
• Compile-time safety
• Centralized authorization control

---

Query execution with projection

public async Task<IDictionary<string, object>> GetCustomerAsync(
    Guid id,
    IReadOnlyCollection<string> fields,
    CancellationToken cancellationToken)
{
    var projections = fields
        .Select(f => CustomerFields.Map[f])
        .ToArray();

    var query = _db.Customers
        .Where(c => c.Id == id)
        .Select(BuildProjection(projections));

    return await query.SingleAsync(cancellationToken);
}

Projection builder:

private static Expression<Func<Customer, IDictionary<string, object>>> BuildProjection(
    Expression<Func<Customer, object>>[] selectors)
{
    return customer => selectors.ToDictionary(
        s => GetFieldName(s),
        s => s.Compile().Invoke(customer)
    );
}

Key properties:

• No full entity materialization
• Projection happens at query level
• Field list drives execution

---

Serialization

The returned dictionary is serialized directly via System.Text.Json.

No custom converters. No dynamic objects. No reflection at runtime.

---

6. Design trade-offs

Approach	Pros	Cons
Fixed DTOs	Simple	DTO explosion, poor evolution
Multiple endpoints	Explicit	Maintence overhead
GraphQL	Powerful	Operational and cognitive cost
Field selection (this)	Efficient, REST-native	More backend complexity

What you gain:

• Payload reduction
• Better cache utilization
• Easier API evolution
• Explicit client intent

What you give up:

• Simplicity of static DTOs
• Some tooling familiarity
• Slightly higher implementation complexity

What you implicitly accept:

• Clients must be disciplined
• Documentation becomes critical
• Validation logic is unavoidable

---

7. Common mistakes and misconceptions

"We can just ignore unrequested fields during serialization"

Why it happends:

• Looks simpler

Problem:

• Database and domain layers still over-fetch

Avoidance:

• Always push projection down to the query

---

"Let clients request any field dynamically"

Why it happends:

• Over-flexibility mindset

Problem:

• Security leaks
• Contract instability

Avoidance:

• Explicit server-side field registry

---

"Missing fields should be returned as null"

Why it happends:

• Misunderstanding semantics

Problem:

• Breaks client assumptions
• Blurs meaning

Avoidance:

• Omitted means omitted

---

8. Operational and production considerations

Things to monitor:

• Payload size distribution
• p95/p99 latency by field set
• Cache hit ratio variance
• GC pressure from allocations

What degrades first:

• Serialization cost under high cardinality field sets
• Cache efficiency if field sets are too granula

Operational risks:

• Unbounded combinations of field sets
• Cache fragmentation
• Poor documentation causing misuse

Mitigations:

• Field set normalization
• Max field count limits
• Default field sets for common clients

---

9. When NOT to use this

Do not use sparse fieldsets when:

• You control both client and server tightly
• Payload size is negligible
• API is strictly internal and short-lived
• Regulatory requirements mandate fixed schemas
• Team maturity does not support contract discipline

In these cases, simpler DTO-based APIs are safer.

---

10. Key takeaways

• Field selection is about evolution, not just performance
• Over-fetching is a hidden cost that compounds over time
• Projections must influence data access, not just serialization
• Explicit field contracts are mandatory for safety
• Cache behavior must be considered early
• Flexibility without constraints becomes technical debt
• REST can evolve without becoming GraphQL

---

11. High-Level Overview

Visual representation of the end-to-end flow, highlighting the transactional boundary, outbox persistence, asynchronous dispatch, and downstream consumption.

Scroll to zoom • Drag to pan