Skip to content

architecture.md

Latest commit

 

History

History
386 lines (294 loc) · 24.9 KB

architecture.md

File metadata and controls

386 lines (294 loc) · 24.9 KB
Raw

PBJ Architecture

This document describes the high-level architecture of PBJ — its module structure, component relationships, data flows, and key design decisions.

For protobuf specification coverage, see protobuf-and-schemas.md. For compiler internals, see code-generation.md. For codec details, see codecs.md, codec-protobuf.md, and codec-json.md.

System Overview

PBJ has three major subsystems:

  1. Compiler — a Gradle plugin that parses .proto files and generates Java source code
  2. Runtime — the library that generated code depends on at runtime (codecs, IO abstractions, collection types)
  3. gRPC — a client and server implementation built on Helidon's HTTP/2 stack, with no io.grpc dependency
┌─────────────────────────────────────────────────────────────┐
│                      Build Time                             │
│  ┌────────────────┐    .proto files                         │
│  │  pbj-compiler  │ ──────────────────► Generated Java      │
│  │ (Gradle Plugin)│                     (models, codecs,    │
│  └────────────────┘                     schemas, tests)     │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                      Runtime                                │
│                                                             │
│  ┌──────────────┐   ┌──────────────┐   ┌─────────────────┐  │
│  │ pbj-runtime  │◄──│ Generated    │──►│  Application    │  │
│  │ (codecs, IO, │   │ Code         │   │  Code           │  │
│  │  types)      │   └──────────────┘   └────────┬────────┘  │
│  └──────┬───────┘                               │           │
│         │                                       │           │
│  ┌──────┴────────────────────────────────────┐  │           │
│  │              gRPC Layer                   │◄─┘           │
│  │  ┌──────────────┐  ┌───────────────────┐  │              │
│  │  │ pbj-grpc-    │  │ pbj-grpc-client-  │  │              │
│  │  │ helidon      │  │ helidon           │  │              │
│  │  │ (server)     │  │ (client)          │  │              │
│  │  └──────┬───────┘  └────────┬──────────┘  │              │
│  │         └────────┬──────────┘             │              │
│  │           ┌──────┴───────┐                │              │
│  │           │ pbj-grpc-    │                │              │
│  │           │ common       │                │              │
│  │           │ (compression)│                │              │
│  │           └──────────────┘                │              │
│  └───────────────────────────────────────────┘              │
└─────────────────────────────────────────────────────────────┘

Module Structure

PBJ is organized as two independent top-level Gradle projects. All modules under pbj-core use the Java Platform Module System (JPMS), except the compiler (which is a Gradle plugin and cannot be a JPMS module).

pbj-core/ — Main Library

Module JPMS Name Purpose
pbj-compiler (none — Gradle plugin) Parses .proto files, generates Java source code
pbj-runtime com.hedera.pbj.runtime Core runtime: codecs, IO abstractions, collection types, gRPC interfaces
pbj-grpc-helidon com.hedera.pbj.grpc.helidon gRPC and gRPC-Web server on Helidon WebServer
pbj-grpc-helidon-config com.hedera.pbj.grpc.helidon.config Helidon annotation processor config for gRPC server
pbj-grpc-client-helidon com.hedera.pbj.grpc.client.helidon gRPC client using Helidon HTTP/2 WebClient
pbj-grpc-common com.hedera.pbj.grpc.common Shared gRPC utilities (framing, timeouts, compression transformers)
hiero-dependency-versions/ — BOM for dependency version management

pbj-integration-tests/ — Validation Suite

A separate Gradle project that downloads the Hiero protobufs (tag v0.55.0) and generates code with both PBJ and Google protoc. Runs 100k+ tests to validate binary compatibility, plus JMH benchmarks and fuzz tests.

Dependency Graph

pbj-compiler (build-time only, not a runtime dependency)

pbj-runtime
├── org.antlr:antlr4-runtime (JSON parser)
└── jdk.unsupported (Unsafe for optimized byte copies)

pbj-grpc-common
├── pbj-runtime
└── com.github.luben:zstd-jni (Zstandard compression)

pbj-grpc-helidon (server)
├── pbj-runtime
├── pbj-grpc-common
├── pbj-grpc-helidon-config
└── io.helidon.webserver.http2

pbj-grpc-client-helidon (client)
├── pbj-runtime
├── pbj-grpc-common
└── io.helidon.webclient.http2

The compiler has no runtime dependency on pbj-runtime — it generates code that imports runtime types, but the compiler itself only needs ANTLR and Gradle APIs.

Compiler Architecture

The compiler is a Gradle plugin (plugin ID: com.hedera.pbj.pbj-compiler) that transforms .proto schema files into Java source code. It runs at build time as part of the Gradle compilation pipeline.

Compilation Pipeline

.proto files (source + classpath JARs)
         │
         ▼
┌─────────────────────┐
│  Phase 1: Scan      │  Parse ALL proto files to build global
│  (LookupHelper)     │  type/package resolution tables
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│  Phase 2: Parse     │  ANTLR lexer + parser per source file
│  (Protobuf3.g4)     │  → parse tree (not full AST)
└─────────┬───────────┘
          │
          ▼
┌─────────────────────┐
│  Phase 3: Generate  │  Walk parse tree, run generators
│  (Generators)       │  per top-level definition
└─────────┬───────────┘
          │
          ▼
    Generated Java source
    (models, codecs, schemas, tests)

Generated Artifacts

For each protobuf message, the compiler produces up to five Java files (model, schema, protobuf codec, JSON codec, and test). Enums and services each produce a single file. For full details on the generators and output structure, see code-generation.md.

Field Model

The compiler uses lightweight field records (SingleField, OneOfField, MapField) extracted directly from ANTLR parse tree contexts, rather than a full AST. For details, see code-generation.md.

Runtime Library

The runtime (pbj-runtime) provides everything that generated code needs at runtime. It has four main areas:

Codec Interfaces

The Codec<T> interface is the core serialization contract. Every generated message exposes singleton PROTOBUF and JSON codec instances. Codecs provide parse, write, measure, and fast-equals operations with configurable safety limits (maxSize and maxDepth). For full details, see codecs.md.

IO Abstractions

Codecs read and write through abstract sequential data interfaces (ReadableSequentialData, WritableSequentialData), decoupling serialization from the underlying byte source. Concrete implementations include BufferedData (byte buffers), ReadableStreamingData (input streams), WritableStreamingData (output streams), and Bytes (immutable byte sequences). For details, see codecs.md.

Collection Types

PBJ provides specialized collection types optimized for generated code:

  • PbjMap<K, V> — an immutable map wrapper that exposes getSortedKeys() for deterministic iteration. Maps are always serialized with keys in natural sort order.
  • UnmodifiableArrayList<E> — a list that starts mutable during parsing (elements added one at a time) and is frozen via makeReadOnly() when parsing completes. More efficient than copying into Collections.unmodifiableList().
  • Bytes — an immutable byte sequence. Wraps byte[] or ByteBuffer and prevents accidental mutation. Supports efficient writeTo(MessageDigest) and writeTo(OutputStream) without copying.
  • OneOf<E> — a type-safe discriminated union for protobuf oneof fields. Stores a discriminator enum (kind) and the value. ComparableOneOf<E> adds Comparable support.

Serialization Helpers

Static utility classes provide the low-level read/write primitives that generated codecs call:

  • ProtoParserTools — reads varints, fixed-width values, strings, bytes, and handles unknown field skipping/extraction
  • ProtoWriterTools — writes tags, scalars, length-delimited fields, and repeated fields (with packed encoding)
  • ProtoArrayWriterTools — mirror of ProtoWriterTools that writes directly to byte[] for maximum performance (avoids virtual dispatch)
  • JsonTools — JSON parsing (via ANTLR), field name conversion (snake_case to camelCase), string escaping, and typed value extraction

gRPC Architecture

PBJ implements gRPC directly on Helidon's HTTP/2 stack, without depending on Google's io.grpc library. This provides low-level control over bytes and framing, eliminates a large transitive dependency tree, and allows fail-fast security behavior.

Runtime gRPC Interfaces

The pbj-runtime module defines the gRPC abstractions that both client and server implement:

ServiceInterface              GrpcClient
  - serviceName()              - createCall(method, reqCodec, replyCodec, pipeline)
  - fullName()
  - methods()                 GrpcCall<Req, Reply>
  - open(method, options,       - sendRequest(req, endOfStream)
         pipeline)              - completeRequests()

Pipeline<T> (extends Flow.Subscriber<T>)
  - onNext(T)
  - clientEndStreamReceived()
  - onComplete()
  - onError(Throwable)

Generated service interfaces extend ServiceInterface and provide an open() method that routes to the correct RPC handler based on the method enum.

gRPC Message Framing

Both client and server use the standard gRPC length-prefixed framing:

┌──────────┬──────────────┬─────────────────────┐
│ 1 byte   │ 4 bytes      │ N bytes             │
│ compress │ msg length   │ message payload     │
│ flag     │ (big-endian) │ (raw or compressed) │
└──────────┴──────────────┴─────────────────────┘

Compression is negotiated per-call via HTTP/2 headers:

  • grpc-accept-encoding — compression algorithms the sender supports
  • grpc-encoding — the algorithm used for this message

Server (pbj-grpc-helidon)

The server integrates with Helidon WebServer as an HTTP/2 sub-protocol provider, discovered via ServiceLoader:

Helidon WebServer (HTTP/2)
    │
    ▼ (ServiceLoader)
PbjProtocolProvider
    │
    ▼ (creates per connection)
PbjProtocolSelector
    │
    ▼ (creates per HTTP/2 stream)
PbjProtocolHandler
    ├── Parses HTTP/2 headers (:method POST, :path /service/method)
    ├── Reads content-type, grpc-encoding, grpc-timeout headers
    ├── Buffers and deframes DATA frames
    ├── Decompresses payload if needed
    ├── Routes to ServiceInterface via PbjServiceRoute → PbjMethodRoute
    ├── Deserializes request via Codec
    ├── Invokes service method
    ├── Serializes response via Codec
    ├── Frames and compresses response
    └── Writes HTTP/2 DATA frames + trailers (grpc-status)

The server supports all four gRPC call types (unary, client-streaming, server-streaming, bidirectional) and handles deadline propagation via the grpc-timeout header.

PbjProtocolSelector declines requests with an application/grpc-web* content type, so Helidon passes them on to the HttpRouting. There, PbjGrpcWebService serves gRPC-Web over HTTP/1.1 and HTTP/2: it reads the request body, deframes it (decoding base64 first in text mode), calls the same ServiceInterface.open(...), and writes the replies as frames followed by a final trailers frame flagged 0x80, instead of HTTP/2 trailers. It also answers CORS preflight requests. Both handlers share the transport-neutral framing code in pbj-grpc-common (GrpcDeframer, GrpcFramer, Base64ChunkDecoder). Server request options and timeout parsing live in pbj-grpc-helidon. The client uses the same GrpcDeframer through a bounded message queue, preserving wire compression flags until response processing.

Native gRPC and gRPC-Web are registered separately because Helidon routes them through different protocol APIs. Share the same PbjConfig and PbjGrpcServiceConfig instances explicitly so message limits and compression settings stay aligned:

final PbjConfig protocolConfig = PbjConfig.builder()
        .maxMessageSizeBytes(4 * 1024 * 1024)
        .build();
final PbjGrpcServiceConfig serviceConfig = PbjGrpcServiceConfig.DEFAULT;
WebServer.builder()
        .addProtocol(protocolConfig)
        .addRouting(PbjRouting.builder().service(service, serviceConfig))
        .addRouting(HttpRouting.builder().register(PbjGrpcWebService.builder(service)
                .config(protocolConfig)
                .serviceConfig(serviceConfig)
                .build()))
        .build();

The web service owns its admission and lifetime limits. defaultTimeout(Duration) is five minutes when the client omits a valid grpc-timeout; maxTimeout(Duration) clamps both client and default lifetimes to one hour by default. maxConcurrentCalls(int) defaults to 1024 admitted calls per PbjGrpcWebService instance, shared across that instance's methods, and excess calls fail with RESOURCE_EXHAUSTED. Configure these values explicitly for long-lived subscriptions and reconnect subscriptions when their maximum lifetime expires. They are separate from the listener's connection, HTTP/2 stream, concurrent-request, and idle-timeout limits.

Request metadata excludes protocol, routing, and hop-by-hop headers. Authorization and custom application metadata remain available. Cookies are omitted unless forwardCookies(true) is selected; applications that opt into cookie authentication must provide their own CSRF protection. Origin allowlists accept serialized HTTP(S) origins or *; paths, credentials, query strings, fragments, and the opaque null origin are rejected.

Cancellation records a terminal outcome independently of response writes and interrupts the dispatch worker and active response writers. Signals delivered to the application request pipeline are serialized, so onError is deferred until an active onNext callback returns. Interruption is cooperative: application callbacks and application-owned asynchronous publishers must release resources when cancelled; PBJ cannot forcibly stop application code that ignores interruption. Helidon's HttpService API does not expose an idle peer disconnect or HTTP/2 stream reset, so these server-owned lifetimes also bound otherwise idle calls after a client leaves.

Method counters use a transport tag with grpc or grpc-web so failures and deadlines can be attributed to the correct endpoint. Dashboard queries that select individual time series must include this tag; aggregate queries can sum over both transports. Zstd is installed once by a shared registry initializer before either server transport negotiates compression, including a server that only registers gRPC-Web.

Native gRPC deadline scheduling uses the remaining timeout duration, rather than treating an absolute System.nanoTime() value as a delay. This is a behavior change: a client-specified grpc-timeout now terminates long-lived native streams when it expires. Completion and terminal error writes are serialized with in-flight response messages. Consumers must verify their consensus-node and block-node client timeout settings during integration; the PBJ repository tests do not constitute an end-to-end consensus-node validation.

The dependency platform selects Helidon 4.5.6. This patch line contains HTTP/2 hardening, including limits on header-table growth and connection-window accounting; see the Helidon release notes and Oracle October 2026 advisory. Applications that manage Helidon versions independently must select a compatible patched version as well. PBJ's call limits complement the server's connection, concurrent-request, HTTP/2 stream, and idle-timeout limits; configure those for the deployment rather than relying on browser-provided timeout headers.

Client (pbj-grpc-client-helidon)

The client wraps Helidon's HTTP/2 WebClient:

PbjGrpcClient (implements GrpcClient)
    │
    ▼ (per call)
PbjGrpcCall<Req, Reply> (implements GrpcCall)
    ├── sendRequest(req, endOfStream)
    │   ├── Serializes via Codec
    │   ├── Compresses if negotiated
    │   ├── Frames as gRPC message
    │   └── Sends HTTP/2 DATA frame
    │
    └── Receives responses
        ├── PbjGrpcDatagramReader deframes DATA
        ├── Decompresses payload
        ├── Deserializes via Codec
        ├── Calls Pipeline.onNext(reply)
        └── On trailers: checks grpc-status, calls onComplete/onError

The client supports custom metadata (propagated as HTTP/2 headers), TLS configuration, and connection pooling via the underlying WebClient.

Compression (pbj-grpc-common)

Compression algorithms are registered in a global GrpcCompression registry. Built-in support:

Algorithm Header value Implementation
Identity (none) identity IdentityGrpcTransformer
Gzip gzip GzipGrpcTransformer
Zstandard zstd ZstdGrpcTransformer (in pbj-grpc-common, uses zstd-jni)

Custom compression algorithms can be registered at runtime via GrpcCompression.registerCompressor() and registerDecompressor(). Inbound compressed messages use Decompressor.decompress(Bytes, int maxOutputSize): custom decompressors must implement this bounded overload and stop before buffering output beyond the supplied limit. Implementing only the legacy decompress(Bytes) method remains source-compatible, but bounded inbound decoding fails with UNIMPLEMENTED until the algorithm supplies a bounded implementation. Gzip and zstd enforce the limit while streaming their output; exceeding it fails with RESOURCE_EXHAUSTED rather than allocating the complete expanded payload.

Data Flow: End to End

Proto to Running Service

 1. Developer writes .proto file
                │
 2. Gradle build triggers PbjCompilerTask
                │
 3. Compiler generates: Model + ProtoCodec + JsonCodec + Schema + ServiceInterface
                │
 4. Java compiler compiles generated + hand-written code
                │
 5. Application creates ServiceInterface implementation
                │
 6. Helidon WebServer starts, discovers PbjProtocolProvider
                │
 7. Client sends gRPC request (HTTP/2 POST)
                │
 8. PbjProtocolHandler deframes → Codec.parse() → ServiceInterface.open()
                │
 9. Service method returns response
                │
10. Codec.write() → PbjProtocolHandler frames → HTTP/2 response

Serialization Round-Trip

Model object (e.g., HelloRequest)
    │
    ▼  Codec.write(obj, output)
WritableSequentialData ───► bytes on wire
    │                        (fields in ascending order,
    │                         maps sorted by key,
    │                         default fields omitted)
    │
    ▼  Codec.parse(input)
ReadableSequentialData ───► Model object
                            (unknown fields optionally preserved,
                             missing fields → null)

Key Design Decisions

Immutable Model Objects

Generated model classes are immutable — all fields are final, there are no setters, and construction goes through a Builder. This enables safe sharing across threads without synchronization, which is critical in multi-threaded code.

Separation of Concerns

Each generated message produces four distinct classes (model, schema, proto codec, JSON codec) rather than bundling serialization logic into the model. This keeps each class focused, avoids polluting the model API with serialization details, and allows different codecs to be optimized independently.

No io.grpc Dependency

PBJ implements the gRPC wire protocol directly on Helidon's HTTP/2 stack. This eliminates io.grpc's large transitive dependency tree, gives PBJ full control over byte handling and framing, and enables features like custom compression and fail-fast security behavior that are difficult to achieve through io.grpc's abstractions.

Two Write Paths

Generated codecs provide both a streaming write path (write(T, WritableSequentialData)) and a byte-array write path (write(T, byte[], offset)). The array path avoids virtual dispatch overhead on the output stream and is measurably faster for small-to-medium messages. The streaming path handles cases where the output destination is not a simple byte array (network sockets, direct buffers).

ANTLR for Both Proto and JSON Parsing

PBJ uses ANTLR4 for parsing both .proto schema files (at build time) and JSON data (at runtime). For proto files this is the natural choice. For JSON, the ANTLR approach builds a parse tree before walking it, which is simpler than streaming JSON parsing but means the entire JSON payload must fit in memory. This is acceptable for PBJ's target use case where messages are bounded by maxSize.

Lazy Computation

Model objects defer expensive computations (hashCode(), protobufEncodedSize()) until first access, then cache the result. This avoids paying the cost in constructors — important when objects are created in hot paths but may never be hashed or measured. The caching uses a benign-race idiom (safe because the computation is idempotent and the fields are int).

Technology Stack

Component Technology Version
Language Java (Temurin) 25
Module system JPMS —
Build Gradle (Kotlin DSL) 9.1.0
Proto parsing ANTLR 4.13.2
HTTP/2 server Helidon WebServer 4.5.6
HTTP/2 client Helidon WebClient 4.5.6
Compression zstd-jni —
Testing JUnit 5, Mockito, AssertJ —
Benchmarks JMH —