Purpose: Enable developers to flatten nested message fields to the parent level in JSON output. With flatten = true, child message fields are promoted to the parent object. With flatten_prefix, a prefix is prepended to avoid name collisions (e.g., billing_street, shipping_street). Single-level flatten only (no recursive flattening).
Output: Go generators produce custom MarshalJSON/UnmarshalJSON for messages with flatten-annotated fields. TypeScript interfaces include promoted child fields at the parent level. OpenAPI uses allOf to represent the flattened structure. Golden tests cover basic flatten, prefix flatten, and dual-flatten scenarios.
<execution_context> @/Users/sebastienmelki/.claude/get-shit-done/workflows/execute-plan.md @/Users/sebastienmelki/.claude/get-shit-done/templates/summary.md </execution_context>
Existing encoding patterns to follow (established in Phase 4-6)
@internal/httpgen/encoding.go @internal/httpgen/nullable.go @internal/httpgen/timestamp_format.go
Annotation functions from Plan 01#
@internal/annotations/flatten.go
Generator entry points to modify#
@internal/httpgen/generator.go @internal/clientgen/generator.go @internal/tsclientgen/generator.go @internal/tsclientgen/types.go @internal/openapiv3/generator.go @internal/openapiv3/types.go
- Define context structs:
type FlattenContext struct {
Message *protogen.Message
FlattenInfos []*FlattenFieldInfo
}
type FlattenFieldInfo struct {
Field *protogen.Field
Prefix string // flatten_prefix value (may be empty)
}- Implement detection and collection functions:
hasFlattenFields(message) bool-- checks if any field has flatten annotationgetFlattenFieldInfos(message) []*FlattenFieldInfo-- returns flatten fields with their prefixescollectFlattenContexts(file) []*FlattenContext-- collects from all messages in filevalidateFlattenAnnotations(file) error-- validates all flatten fields in file- For each message, for each field: call
annotations.ValidateFlattenField - For each message with flatten fields: call
annotations.ValidateFlattenCollisions - Check for MarshalJSON conflict: if message has BOTH flatten AND any of (int64 NUMBER encoding, nullable, empty_behavior, timestamp_format non-default, bytes_encoding non-default, oneof_config), emit error: "message X has both flatten and [other feature] on the same message -- this combination is not yet supported"
- For each message, for each field: call
- Implement
generateFlattenFile(file) errormethod on Generator:
- File naming:
file.GeneratedFilenamePrefix + "_flatten.pb.go" - Only generate if there are messages with flatten annotations
- Run validation first; return error if validation fails
- Generated MarshalJSON pattern:
func (x *MessageName) MarshalJSON() ([]byte, error) {
if x == nil {
return []byte("null"), nil
}
data, err := protojson.Marshal(x)
if err != nil {
return nil, err
}
var raw map[string]json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return nil, err
}
// For each flattened field:
// 1. Remove the nested field from raw
// 2. Marshal the child message (use json.Marshal to respect child's MarshalJSON)
// 3. Add each child field to raw with optional prefix
if x.FlattenedFieldGoName != nil {
delete(raw, "flattenedFieldJsonName")
childData, err := json.Marshal(x.FlattenedFieldGoName)
if err != nil {
return nil, err
}
var childRaw map[string]json.RawMessage
if err := json.Unmarshal(childData, &childRaw); err != nil {
return nil, err
}
for k, v := range childRaw {
raw["prefix"+k] = v // prefix may be empty
}
}
return json.Marshal(raw)
}IMPORTANT: Use json.Marshal(x.FlattenedFieldGoName) to marshal the child, NOT protojson.Marshal(). This ensures that if the child message has its own MarshalJSON (for int64 encoding, timestamps, etc.), it gets invoked. This is the key to annotation composability as stated in CONTEXT.md.
- Generated UnmarshalJSON pattern:
func (x *MessageName) UnmarshalJSON(data []byte) error {
var raw map[string]json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
// For each flattened field:
// 1. Extract child fields from raw into a new map (with prefix stripped)
// 2. Marshal the child map and unmarshal into the child message
// 3. Delete the extracted fields from raw
childRaw := make(map[string]json.RawMessage)
// For each known child field of the flattened message:
// Check if "prefix" + childJsonName exists in raw
if v, ok := raw["prefixChildField1"]; ok {
childRaw["childField1"] = v
delete(raw, "prefixChildField1")
}
if v, ok := raw["prefixChildField2"]; ok {
childRaw["childField2"] = v
delete(raw, "prefixChildField2")
}
// ... for each child field
if len(childRaw) > 0 {
childData, err := json.Marshal(childRaw)
if err != nil {
return err
}
x.FlattenedFieldGoName = &ChildMessageType{}
if err := json.Unmarshal(childData, x.FlattenedFieldGoName); err != nil {
return err
}
}
// Re-marshal remaining and unmarshal via protojson for non-flattened fields
remaining, err := json.Marshal(raw)
if err != nil {
return err
}
return protojson.Unmarshal(remaining, x)
}IMPORTANT for UnmarshalJSON: The child field JSON names must be known at generation time (they come from the child message's field descriptors). Generate a concrete list of field name checks, not a dynamic loop. Use json.Unmarshal(childData, x.FlattenedFieldGoName) to unmarshal the child, respecting its own UnmarshalJSON if present.
-
Required imports for generated code:
encoding/json,google.golang.org/protobuf/encoding/protojson -
In generator.go, add call to
g.generateFlattenFile(file)after oneof_discriminator file call. Order: ..., bytes_encoding, oneof_discriminator, flatten. -
Copy internal/httpgen/flatten.go to internal/clientgen/flatten.go (change package name only). This follows the established identical-file pattern.
-
In clientgen/generator.go, add the same call.
For a message with flattened fields, modify the interface generation:
- When generating the interface for a message with flatten fields, skip emitting the flattened field as a nested property
- Instead, inline the child message's fields at the parent level with the optional prefix
- Example:
// Without flatten:
export interface Order {
id: string;
billing: Address;
shipping: Address;
}
// With flatten + prefix:
export interface Order {
id: string;
billing_street: string;
billing_city: string;
billing_zip: string;
shipping_street: string;
shipping_city: string;
shipping_zip: string;
}Implementation approach:
- In the interface generation function, check each field with
annotations.IsFlattenField(field) - For flatten fields, get the child message and iterate over its fields
- Apply
annotations.GetFlattenPrefix(field)to each child field's JSON name - Emit the child fields at the parent level with their types
OpenAPI (internal/openapiv3/types.go and generator.go):
For messages with flattened fields, modify the schema generation to use allOf:
Order:
allOf:
- type: object
properties:
id:
type: string
- type: object
description: "billing (flattened with prefix billing_)"
properties:
billing_street:
type: string
billing_city:
type: string
billing_zip:
type: string
- type: object
description: "shipping (flattened with prefix shipping_)"
properties:
shipping_street:
type: string
shipping_city:
type: string
shipping_zip:
type: stringAlternative simpler approach: Generate a flat object schema with all properties at the top level (without using allOf). This is simpler and produces equivalent validation:
Order:
type: object
properties:
id:
type: string
billing_street:
type: string
billing_city:
type: string
billing_zip:
type: string
shipping_street:
type: string
shipping_city:
type: string
shipping_zip:
type: stringChoose the approach that best matches the existing OpenAPI generation patterns. If the codebase builds schemas property-by-property, the flat approach is easier. If it composes schemas, allOf is more semantic.
Golden test proto (flatten.proto):
Create the test proto in httpgen/testdata/proto/ and symlink to other generators:
syntax = "proto3";
package flatten.test;
import "sebuf/http/annotations.proto";
option go_package = "github.com/SebastienMelki/sebuf/internal/httpgen/testdata/flatten";
// Child message used for flattening
message Address {
string street = 1;
string city = 2;
string zip = 3;
}
message ContactInfo {
string email = 1;
string phone = 2;
}
// Basic flatten without prefix
message SimpleFlatten {
string id = 1;
Address address = 2 [(sebuf.http.flatten) = true];
}
// Flatten with prefix (two flattened fields of same type)
message DualFlatten {
string id = 1;
Address billing = 2 [
(sebuf.http.flatten) = true,
(sebuf.http.flatten_prefix) = "billing_"
];
Address shipping = 3 [
(sebuf.http.flatten) = true,
(sebuf.http.flatten_prefix) = "shipping_"
];
}
// Flatten with mixed fields (some flattened, some not)
message MixedFlatten {
string id = 1;
Address address = 2 [(sebuf.http.flatten) = true];
ContactInfo contact = 3; // Not flattened
string notes = 4;
}
// No flatten annotation (backward compatible)
message PlainNested {
string id = 1;
Address address = 2;
}
service FlattenService {
option (sebuf.http.service_config) = {
base_path: "/api/v1"
};
rpc TestSimpleFlatten(SimpleFlatten) returns (SimpleFlatten) {
option (sebuf.http.config) = {
path: "/flatten/simple"
method: HTTP_METHOD_POST
};
}
rpc TestDualFlatten(DualFlatten) returns (DualFlatten) {
option (sebuf.http.config) = {
path: "/flatten/dual"
method: HTTP_METHOD_POST
};
}
rpc TestMixedFlatten(MixedFlatten) returns (MixedFlatten) {
option (sebuf.http.config) = {
path: "/flatten/mixed"
method: HTTP_METHOD_POST
};
}
rpc TestPlainNested(PlainNested) returns (PlainNested) {
option (sebuf.http.config) = {
path: "/flatten/plain"
method: HTTP_METHOD_POST
};
}
}Create symlinks from clientgen, tsclientgen, openapiv3 testdata/proto/ directories to the httpgen source proto.
Run UPDATE_GOLDEN=1 tests to generate golden files, then verify they pass without the flag:
UPDATE_GOLDEN=1 go test -run TestExhaustiveGoldenFiles ./internal/httpgen/...
UPDATE_GOLDEN=1 go test -run TestGoldenFiles ./internal/clientgen/...
UPDATE_GOLDEN=1 go test -run TestGoldenFiles ./internal/tsclientgen/...
UPDATE_GOLDEN=1 go test -run TestExhaustiveGoldenFiles ./internal/openapiv3/...<success_criteria>
- SimpleFlatten promotes Address fields (street, city, zip) to parent level in JSON
- DualFlatten with prefixes produces billing_street, billing_city, shipping_street, shipping_city
- MixedFlatten shows both flattened and non-flattened fields correctly
- go-http and go-client produce identical code (byte-level after normalization)
- TypeScript interfaces correctly inline flattened fields with prefixes
- OpenAPI schemas accurately represent the flattened wire format
- Generation-time error on flatten conflicts (collisions, invalid field types, MarshalJSON conflicts)
- All golden tests pass, all existing tests pass </success_criteria>
