Purpose: Enable developers to choose how bytes fields serialize to JSON. BASE64 (default), BASE64_RAW, BASE64URL, BASE64URL_RAW, and HEX are supported. Go server and client must produce identical JSON. OpenAPI must document the actual encoding format.
Output: Go generators produce custom MarshalJSON/UnmarshalJSON for non-default bytes encodings using the protojson-base-then-modify-map pattern. OpenAPI schema generation accounts for bytes encoding annotations with appropriate format and pattern fields.
<execution_context> @/Users/sebastienmelki/.claude/get-shit-done/workflows/execute-plan.md @/Users/sebastienmelki/.claude/get-shit-done/templates/summary.md </execution_context>
Existing encoding patterns to follow exactly
@internal/httpgen/encoding.go @internal/httpgen/nullable.go @internal/clientgen/encoding.go
Annotation functions from Plan 01#
@internal/annotations/bytes_encoding.go
Generator entry points to modify#
@internal/httpgen/generator.go @internal/clientgen/generator.go
Type mapping files#
@internal/tsclientgen/types.go @internal/openapiv3/types.go
- Define BytesEncodingContext struct:
type BytesEncodingContext struct {
Message *protogen.Message
BytesFields []*BytesEncodingFieldInfo
}
type BytesEncodingFieldInfo struct {
Field *protogen.Field
Encoding http.BytesEncoding
}- Implement detection functions:
hasBytesEncodingFields(message) bool-- checks if any bytes field has non-default encodinggetBytesEncodingFields(message) []*BytesEncodingFieldInfo-- returns annotated fields with their encodingscollectBytesEncodingContext(file) []*BytesEncodingContext-- recursively collects from all messagescollectBytesEncodingMessages(messages, *contexts)-- recursive helpervalidateBytesEncodingAnnotations(file) error-- validates all fields in file
- Implement
generateBytesEncodingFile(file) errormethod on Generator:
- File naming:
file.GeneratedFilenamePrefix + "_bytes_encoding.pb.go" - Only generate if there are messages with non-default bytes encoding fields
- Use protojson for base serialization, then modify the map
- Generated MarshalJSON pattern -- uses Go standard library encoding packages:
func (x *MessageName) MarshalJSON() ([]byte, error) {
if x == nil {
return []byte("null"), nil
}
data, err := protojson.Marshal(x)
if err != nil {
return nil, err
}
var raw map[string]json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return nil, err
}
// For HEX:
if len(x.FieldName) > 0 {
raw["fieldJsonName"], _ = json.Marshal(hex.EncodeToString(x.FieldName))
}
// For BASE64_RAW:
if len(x.FieldName) > 0 {
raw["fieldJsonName"], _ = json.Marshal(base64.RawStdEncoding.EncodeToString(x.FieldName))
}
// For BASE64URL:
if len(x.FieldName) > 0 {
raw["fieldJsonName"], _ = json.Marshal(base64.URLEncoding.EncodeToString(x.FieldName))
}
// For BASE64URL_RAW:
if len(x.FieldName) > 0 {
raw["fieldJsonName"], _ = json.Marshal(base64.RawURLEncoding.EncodeToString(x.FieldName))
}
return json.Marshal(raw)
}- Generated UnmarshalJSON pattern:
func (x *MessageName) UnmarshalJSON(data []byte) error {
var raw map[string]json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return err
}
// For HEX: decode hex, re-encode as standard base64, replace in map
if v, ok := raw["fieldJsonName"]; ok {
var s string
if err := json.Unmarshal(v, &s); err == nil {
decoded, err := hex.DecodeString(s)
if err == nil {
raw["fieldJsonName"], _ = json.Marshal(base64.StdEncoding.EncodeToString(decoded))
}
}
}
// For BASE64_RAW: decode with RawStdEncoding, re-encode with StdEncoding
// For BASE64URL: decode with URLEncoding, re-encode with StdEncoding
// For BASE64URL_RAW: decode with RawURLEncoding, re-encode with StdEncoding
data, err := json.Marshal(raw)
if err != nil {
return err
}
return protojson.Unmarshal(data, x)
}- Key mapping of encoding to Go standard library:
- BASE64 -> base64.StdEncoding (default, no custom code needed)
- BASE64_RAW -> base64.RawStdEncoding
- BASE64URL -> base64.URLEncoding
- BASE64URL_RAW -> base64.RawURLEncoding
- HEX -> hex.EncodeToString / hex.DecodeString
-
Required imports for generated code:
encoding/json,encoding/base64,encoding/hex,google.golang.org/protobuf/encoding/protojson -
In generator.go, add call to
g.generateBytesEncodingFile(file)after the timestamp format call. -
Add bytes_encoding validation in generateFile, before the encoding file generation (similar to validateEnumAnnotationsInFile).
-
Copy internal/httpgen/bytes_encoding.go to internal/clientgen/bytes_encoding.go (change package name only, following D-04-02-03).
-
In clientgen/generator.go, add the same calls.
CRITICAL NOTE from RESEARCH:
- Only replace field in the map if the Go field is non-empty (
len(x.Field) > 0). Empty bytes fields are omitted by protojson, so no action needed.
go build ./internal/httpgen/... ./internal/clientgen/...
go test -v ./internal/httpgen/... ./internal/clientgen/...
make lint-fixBytes fields with any encoding variant always serialize to a string in JSON, so the TypeScript type is always string regardless of encoding. The current case protoreflect.BytesKind: return tsString already handles this correctly. No change needed for TypeScript type mapping.
However, add a comment near the BytesKind case documenting that all bytes_encoding variants produce strings (for future maintainability).
OpenAPI (internal/openapiv3/types.go):
In the BytesKind handling, BEFORE or REPLACING the existing generic format: byte handling, add bytes_encoding detection:
case protoreflect.BytesKind:
schema.Type = []string{"string"}
encoding := annotations.GetBytesEncoding(field)
switch encoding {
case http.BytesEncoding_BYTES_ENCODING_HEX:
schema.Format = "hex"
schema.Pattern = "^[0-9a-fA-F]*$"
case http.BytesEncoding_BYTES_ENCODING_BASE64_RAW:
schema.Format = "byte"
schema.Description = "Base64 encoded without padding"
case http.BytesEncoding_BYTES_ENCODING_BASE64URL:
schema.Format = "base64url"
case http.BytesEncoding_BYTES_ENCODING_BASE64URL_RAW:
schema.Format = "base64url"
schema.Description = "URL-safe base64 encoded without padding"
default:
// UNSPECIFIED, BASE64 -> standard base64 with padding
schema.Format = "byte"
}Golden test protos:
Create bytes_encoding.proto in all 4 generator testdata directories. Use symlinks from clientgen/tsclientgen/openapiv3 to httpgen (following established pattern):
syntax = "proto3";
package bytes_encoding.test;
import "sebuf/http/annotations.proto";
option go_package = "github.com/SebastienMelki/sebuf/internal/httpgen/testdata/bytes_encoding";
message BytesEncodingTest {
// Default (BASE64)
bytes default_data = 1;
// Explicit BASE64
bytes base64_data = 2 [(sebuf.http.bytes_encoding) = BYTES_ENCODING_BASE64];
// BASE64_RAW (no padding)
bytes base64_raw_data = 3 [(sebuf.http.bytes_encoding) = BYTES_ENCODING_BASE64_RAW];
// BASE64URL
bytes base64url_data = 4 [(sebuf.http.bytes_encoding) = BYTES_ENCODING_BASE64URL];
// BASE64URL_RAW
bytes base64url_raw_data = 5 [(sebuf.http.bytes_encoding) = BYTES_ENCODING_BASE64URL_RAW];
// HEX
bytes hex_data = 6 [(sebuf.http.bytes_encoding) = BYTES_ENCODING_HEX];
}
service BytesEncodingService {
option (sebuf.http.service_config) = {
base_path: "/api/v1"
};
rpc TestBytesEncoding(BytesEncodingTest) returns (BytesEncodingTest) {
option (sebuf.http.config) = {
path: "/bytes-encoding"
method: HTTP_METHOD_POST
};
}
}Run UPDATE_GOLDEN=1 tests to generate golden files, then verify they pass without the flag:
UPDATE_GOLDEN=1 go test -run TestExhaustiveGoldenFiles ./internal/httpgen/...
UPDATE_GOLDEN=1 go test -run TestGoldenFiles ./internal/clientgen/...
UPDATE_GOLDEN=1 go test -run TestGoldenFiles ./internal/tsclientgen/...
UPDATE_GOLDEN=1 go test -run TestExhaustiveGoldenFiles ./internal/openapiv3/...<success_criteria>
- HEX bytes encoding produces lowercase hex string in Go JSON output
- BASE64_RAW removes padding from base64 output
- BASE64URL uses URL-safe characters
- BASE64URL_RAW combines URL-safe and no-padding
- UnmarshalJSON correctly converts all variants back to standard base64 before protojson
- go-http and go-client produce identical code (byte-level after normalization)
- OpenAPI schemas accurately document the encoding format
- All golden tests pass, all existing tests pass </success_criteria>