Architecture

How Etorm is put together.

If you want to know what actually happens when you run kubectl apply, or where your packets go, this is the page. Five drawings, each with a few notes on its limits and on what happens when a part of it fails.

Control plane

Every service is a set of resources in one Kubernetes-compatible API. You write what you want into a resource's spec. A controller does the work and writes what happened into status.

Control plane: one request
kubectl OpenTofu Console HTTP 1 PROJECT ENDPOINT · <PROJECT>-API.ETORM.COM 2 Authenticate token or session Authorise role, else denied Admit defaults, validation Store sealed before written 3 4 5 Audit log changes and refusals, per project 6 Controllers watch, reconcile 7 watch status
  1. Clients. kubectl, OpenTofu, plain HTTP and the console use the same API. There is no second interface behind it.
  2. Project endpoint. Each project is a logical cluster on its own hostname, with its own objects, roles and audit log.
  3. Authenticate, then authorise. Every request carries an identity and is checked against roles. Without a matching role the answer is no.
  4. Admit. Defaults are filled in and the object is validated before it is accepted. References to other resources must exist.
  5. Store. Records are sealed with an authenticated cipher before they are stored. Tokens and keys are stored hashed or not at all.
  6. Audit log. Creates, updates, deletes and refusals, with who and when. Request bodies are never recorded.
  7. Controllers. They watch resources, act, and report in status.phase and status.conditions. They use the same API as you do.

A project can hold 100 roles and 500 bindings. A person can own up to 5 billing accounts, each with up to 20 projects. The audit log keeps the last 5 000 changes and 1 000 refusals for 90 days.

Controllers look at the current state of resources, not at a queue of events. After an interruption they simply pick up from wherever the resources are, and nothing has to be replayed. Usage isn't counted for time the control plane was down.

Putting everything behind one API means no service gets a private shortcut. Whatever the console can do, kubectl can do, under the same roles. The flip side is that every service depends on the API being up, so that is where we spend most of our care.

More on this: Identity and access, Billing and projects, Security.

Network data plane

A VPC is an Ethernet segment. A member reaches it through an L2 tunnel carried inside WireGuard to an Etorm gateway, which bridges the members of the segment.

Network data plane: one member and its segment
MEMBER HOST 1 etm<vni> L2 device: VXLAN or Geneve wg-etorm WireGuard, its own key WireGuard, UDP frames inside 2 ETORM GATEWAY 3 wg-etorm accepts authenticated peers only Bridge and VTEP per VNI one VNI per VPC 4 Member Member DNS <prefix>::1 DHCP IPv4 only 5 HUB AND SPOKE: EVERY FRAME CROSSES THE GATEWAY OVERLAY MTU ≤ 1350
  1. Member host. A Linux machine with two devices: a WireGuard interface and an L2 device (VXLAN or Geneve) on top of it. The member makes its own key; Etorm only ever holds the public half.
  2. Tunnel. WireGuard over UDP. Inside it, VXLAN (UDP 4789) or Geneve (UDP 6081) carries whole Ethernet frames.
  3. Gateway. Tunnel traffic is accepted only from the addresses of authenticated peers.
  4. Bridge per VNI. One VNI per VPC, from a 24-bit space, never changed once assigned. Tenants are isolated by VNI, so two VPCs can use the same prefix.
  5. On the segment. Other members, a DNS zone on <prefix>::1, and DHCPv4 where the VPC has an IPv4 network.

Each VPC has an IPv6 /64 and, if you ask for it, an IPv4 network from /16 to /29. The overlay MTU is at most 1350 over an IPv6 underlay; the VPC's status tells you the exact value.

Every frame crosses the VPC's gateway, and a VPC has one. If the gateway can't be reached, the members lose the segment until it is back.

We chose hub-and-spoke because it gives one path to debug and one place where isolation is enforced. Layer 2 means broadcast, multicast and discovery work without changes, and the tunnel headers in every packet are what you pay for that.

More on this: Networking, DNS and DHCP.

Compute

A program is a small WebAssembly module. A process is that program plugged into your VPCs with its own Ethernet interfaces. It runs when something happens and stays idle otherwise.

Compute: a process and what wakes it
Program WebAssembly, immutable 1 runs as PROCESS 2 Activation runs, then stops State up to 1 MiB 4 saved on success only Interface MAC, VPC addresses Interface another VPC 5 Frame on an interface Wake over the network Resume from its snapshot 3 wakes ETHERNET INTERFACES ON YOUR VPCS
  1. Program. Checked when it is registered and never changed afterwards. A new version is a new program; point the process at it to roll out.
  2. Process. One program with its own allowance: memory from 64 KiB upwards in powers of two, and a share of a core.
  3. Events. A frame on one of its interfaces wakes it. A sleeping process is woken over the network and resumes from its snapshot.
  4. State. Up to 1 MiB, saved only when an activation succeeds. A failed activation changes nothing.
  5. Interfaces. Each has a MAC and the addresses of its VPC, and shows up as a device of that VPC. The program speaks any protocol it likes.

A failed activation leaves the saved state exactly as it was before. Processes run on a gateway agent, next to the VPCs they are attached to.

A program that only runs on events uses no CPU between them. If a program needs to keep its memory around, it runs as a resident process instead, and it can still sleep to a snapshot in storage when it is idle.

More on this: Compute.

Storage

A bucket holds objects of up to 32 MiB. Each object is sealed with the bucket's own keys before anything is stored, so what is stored shows neither contents nor names.

Storage: one write
Client curl, any HTTP client 1 HTTPS STORAGE 2 Access check role, else denied Seal 64 KiB chunks Spread pieces, no names 3 5 Bucket key 4 Stored sealed pieces only 6 NOTHING LEAVES THIS BOX UNENCRYPTED
  1. Client. PUT, GET, HEAD and DELETE with an API token; list by prefix with a cursor.
  2. Storage. A bucket is a resource like any other, created with kubectl, OpenTofu or the console.
  3. Seal. Objects are sealed in 64 KiB chunks with XChaCha20-Poly1305. Each chunk is bound to its identity, so it cannot be moved or reordered unnoticed.
  4. Bucket key. Every bucket has its own. Delete the bucket and every object in it becomes unreadable at once.
  5. Spread. The sealed pieces are spread before they are stored.
  6. Stored. Only sealed pieces. No public buckets and no anonymous access.

Objects go up to 32 MiB, keys up to 1 024 bytes, and a list call returns up to 200 objects per page.

Because Etorm does the sealing, reading an object takes a role, never a key you have to manage. That is also why there are no presigned URLs or public buckets: every read is an authorised request.

More on this: Storage, Security.

Identity and isolation

Access is denied unless a binding allows it. Tenants are kept apart at three levels: each project has its own API endpoint, each network its own VNI, and each bucket its own key.

Identity and isolation: who reaches what
Person Group Service account tokens, hashed 1 RoleBinding names a role 2 BILLING ACCOUNT · WHO PAYS 3 PROJECT · OWN HOSTNAME, ROLES, AUDIT LOG 4 VPC bounded by its VNI Bucket bounded by its key 5 6 Every other resource bounded by the project denied without a binding
  1. Identities. People, groups and service accounts. A token is shown once and stored only as a hash.
  2. RoleBinding. Gives an identity a role on a project or billing account. An invitation never grants more than the person who made it holds.
  3. Billing account. Says who pays. Roles are inherited from the account to its projects.
  4. Project. The unit of isolation: a separate API endpoint with its own objects, roles and audit log. Deleting it removes everything in it.
  5. VPC. Isolated by its VNI: the gateway bridges only the members of the same VNI.
  6. Bucket. Isolated by its key: every bucket's objects are sealed with that bucket's own keys.

Want to go deeper?

Each product page goes into its own limits, failure modes and design choices. Or write to us and ask the people who built it.