Encoding and Evolution: How Data Survives Schema Changes

The Compatibility Problem

Applications change continuously. Features are added or modified, requirements shift. These changes require corresponding changes to data formats. The complication: in a large system, code changes cannot happen instantaneously. Rolling upgrades deploy new server versions gradually — a few nodes at a time — while client-side applications are at the mercy of users who may not update for months. During any deployment window, multiple versions of the code are running simultaneously, reading and writing to the same databases and message queues. This requires compatibility in two directions: backward compatibility (new code reads data written by old code) and forward compatibility (old code reads data written by new code). Backward compatibility is typically straightforward — you wrote the new code, so you can explicitly handle the old format. Forward compatibility is harder: it requires old code to gracefully ignore fields and structures it has never seen before.

Encoding Formats: A Taxonomy

In-memory data structures use pointers and CPU-optimised layouts. On-disk and on-wire formats must be self-contained byte sequences — no pointers. The translation between the two is encoding/serialisation (to bytes) and decoding/deserialisation (to objects). The choice of encoding format determines: byte size, parse speed, schema evolution capability, cross-language interoperability, and human readability. Note: columnar storage formats like Parquet operate directly on compressed column data without a decode step — a special case called zero-copy encoding.

Protocol Buffers: Field Tags as the Compatibility Mechanism

Protocol Buffers encodes each field as a tag byte (field_number << 3 | wire_type) followed by the value. Field names are absent from the wire format — only tag numbers matter. Strings are length-prefixed UTF-8. Integers use variable-length zigzag encoding (small numbers use fewer bytes). The same record takes 33 bytes in protobuf vs 81 bytes in JSON. Schema evolution rules: you may add new fields (give each a new unique tag number — old readers skip unknown tags using the wire type to determine skip length); you may remove optional fields (never reuse the tag number — reserved tags prevent future reuse accidents); you may rename fields (safe — the wire format never mentions the name). Changing int32 to int64 is safe; narrowing truncates. The repeated modifier encodes lists as repeated occurrences of the same tag — compatible with optional fields, enabling list-to-scalar evolution.

// Protocol Buffers schema (proto3)
message Person {
  string user_name     = 1;
  int64  favorite_num  = 2;
  repeated string interests = 3;
}
// Encoded: 33 bytes. Field names absent — only tag numbers on wire.

// Avro IDL schema
record Person {
  string                   userName;
  union { null, long } favoriteNumber = null;
  array<string>            interests;
}
// Encoded: 32 bytes. No tags — schema must be known to decode.

Apache Avro: Writer and Reader Schemas

Avro's binary encoding contains only values — no type indicators, no field numbers. Decoding is only possible with the exact writer's schema, which the Avro library uses to walk the byte stream field by field. Schema evolution is achieved through writer/reader schema resolution: the library compares the two schemas side-by-side, matching fields by name (not position or tag). Fields present in the writer's schema but absent from the reader's are ignored. Fields present in the reader's schema but absent from the writer's are filled in from the reader's default value. This means: you may only add or remove fields that have default values (otherwise, backward or forward compatibility breaks). Field renaming is backward compatible via aliases in the reader schema. Null values require explicit union types: union { null, long } — enforcing explicit nullability rather than everything-is-nullable-by-default. How does the reader know the writer's schema? It depends on context: large Avro files embed the schema once at the top (object container file format); databases include a schema version number per record and maintain a schema registry; streaming connections negotiate the schema on connection setup.

Schema Evolution Rules by Format

The Fundamental Problems with RPC

RPC (remote procedure call) attempts location transparency: making a remote service call appear syntactically identical to a local function call. This abstraction is fundamentally flawed. A network call and a local call differ in at least five critical ways that cannot be papered over by a framework. The history of enterprise RPC — CORBA, EJB, DCOM, SOAP, WS-* — is a history of failed attempts to hide this distinction. REST succeeds partly because it treats state transfer over the network as a distinct concept from a local function call, making the network boundary explicit.

Four Modes of Dataflow

Compatibility is a relationship between two processes: the one that encodes data and the one that decodes it. The encoding choices and compatibility requirements differ significantly depending on how data flows between those processes. The chapter identifies four principal dataflow modes.

Message Brokers: Decoupling as the Primary Benefit

Message brokers provide four benefits that direct RPC cannot: buffering (absorb producer bursts when consumers are slow), redelivery (automatic retry if a consumer crashes before acknowledging), fan-out (one message delivered to all subscribers of a topic), and service discovery elimination (producers publish to a named topic without knowing consumer addresses). Two distribution patterns: queue (one message → one consumer; work distribution) and topic/pub-sub (one message → all subscribers; event broadcast). Message encoding is the contract — brokers are encoding-agnostic byte carriers. Pairing Avro or Protobuf with a schema registry (Confluent Schema Registry for Kafka, AsyncAPI specifications) lets you enforce and version that contract across producer and consumer deployments. A critical detail: if a consumer re-publishes messages to another topic after processing, it must preserve unknown fields from the original message — otherwise the forward-compatibility problem from Figure 5-1 reappears in the broker pipeline.

Six Mental Models Worth Keeping