Network¶
The network functions the platforms use and how tenants are connected: exchange networks to the Internet and to the corporate network, L2-over-L3 channels and the EVPN fabric, plus the address rules the portal applies. Based on the claugine reference design (docs/network). The data network itself (switches, uplinks, routing to the outside) is designed and run outside the cloud project.
1. L3 layout¶
- Site network — provided by the data network; connects the site to the Internet and to other sites.
- Management gateway — routes the management networks: out-of-band, host management, platform core.
- Payload gateway — connects the exchange networks (IX) to the outside.
- Every landscape is closed the same way: by an external gateway (EGW) in the IX and an internal gateway behind it. Tenants have their own EGW and internal gateway; the provider's published services have
cloud-egw(services network to the IX) andcloud-igw(platform core to the services network).
2. Fabrics and host network stack¶
| Fabric | Carries |
|---|---|
| data | everything except storage: platform, services, transit, VXLAN transport |
| storage | hosts ↔ storage systems |
Both fabrics have no single point of failure; the platform tolerates a network outage of up to one second.
| LAG | Ports | Fabric | Segments |
|---|---|---|---|
| lag1 | NIC1 port 1 + port 2 (25G) | data | host management, platform core, services, exchange networks, VXLAN transport |
| lag2 | NIC2 port 1 + port 2 (25G) | storage | NFS, iSCSI |
LACP, 802.1Q, MTU 9000, plus an out-of-band port. VM networks are Linux bridges created by OpenNebula per network: VLAN on the management platform, VXLAN on the payload platforms. Two NTP servers per site come from the data network.
3. Segments¶
| Kind | Segment | Purpose |
|---|---|---|
| internal | oob | IPMI of all servers |
| internal | host management | host OS management |
| internal | platform core | services that run the platform |
| routed | services (cloud-services) |
services published to users: FE end points, DNS, … |
| routed | tenant main network | delegated to one tenant, /31 and up |
| transit | exchange networks (cloud-ix, corpnet-ix, portal-ix) |
IX to the Internet, IX to a corporate network |
| isolated | VXLAN transport | carries the EVPN overlay between payload hosts |
| isolated | storage | NFS / iSCSI |
| transit | s2s transport | L3 between the legs of L2-over-L3 channels |
| overlay | tenant subnets | created by tenants, VXLAN/EVPN, any addressing |
4. Tenant connections¶
Every tenant has at least one connection; the customer chooses them when ordering:
| Connection | Who may order | Combines with |
|---|---|---|
inet — Internet through the exchange network |
everyone | corpnet, L2 |
corpnet — routed connection to the corporate network |
corporate organizations only | inet, L2 |
l2 — external L2-over-L3 channel |
everyone | inet, corpnet, or alone |
The connectivity record stores the routed connections space-separated (inet, corpnet, corpnet inet), or none when the tenant has only L2 channels.
5. Exchange network to the Internet¶
The IX is a dedicated transit VLAN. Only routers are connected to it, they talk only to each other, nothing outside the VLAN can reach their addresses, and only permitted transit traffic passes.
| Router | Where | Role |
|---|---|---|
| Internet gateways | data network | exchange traffic with the Internet; announce the default route; receive and aggregate the tenants' public subnets |
| Route reflectors | management platform, 3 VyOS VMs | receive the tenants' public subnets from the EGWs and the routes of the Internet gateways; send the full table to all BGP peers |
| EGW — tenant external gateway | payload platform, one VyOS VM per tenant | announces the tenant's public subnets, connected and statically routed |
Tenant side¶
- EGW — controlled by the provider, not reachable by the tenant:
- outside leg in the IX: bandwidth is shaped here to the ordered rate (10–1000 Mbit/s); traffic inside the provider network is not limited;
- inside leg in the tenant's main public network, /31 minimum; the first address is the EGW, the second the tenant's internal gateway (IGW) — the tenant's default gateway is on the EGW;
- lets in and out only the tenant's public addresses (main network and additional public subnets);
- drops addresses not routed in the Internet (RFC 1122, 1918, 3927, 5737, 6598 and other special-purpose blocks).
- Internal gateway (IGW) — the tenant's router between its overlays and the main public network:
vyos(provider-created VyOS),onevr(OpenNebula virtual router) orcustom(the tenant's own appliance); does source NAT for private subnets. Additional public subnets are routed through the IGW. - The main public prefix cannot be changed after the tenant is provisioned.
6. Exchange network to the corporate network¶
The same model as the Internet IX: a dedicated transit VLAN with routers only, corporate network gateways on the data network side, the tenant's EGW (a provider-controlled VyOS VM on the payload platform) with its outside leg in this exchange network and its inside leg in the tenant's landscape, and routes exchanged over BGP through the route reflectors. It connects a tenant to the corporate network instead of, or in addition to, the Internet; internet access of a corpnet-only tenant is controlled by the corporate network.
7. L2-over-L3 channels¶
Extend a tenant L2 segment to a client site over an L3 network: client systems and tenant VMs share one broadcast domain and one subnet. A channel s2s-NNNN has two provider-controlled legs:
leg 1 s2s-NNNN-src |
leg 2 s2s-NNNN-dst |
|
|---|---|---|
| platform | management | payload |
| faces | the client site | the tenant network |
| eth1 | network s2s-NNNN: a VLAN from the provider transport network to the client site, no IP |
the tenant's VXLAN network |
| eth0 | s2s transport network, an address of leg 1 | s2s transport network, an address of leg 2 |
| inside | Linux bridge of eth1 + a VXLAN tunnel (one VNI per channel) | the same |
- The legs exchange the client's Ethernet frames in a VXLAN tunnel over the s2s transport network; neither leg has an address in the client segment.
- The tenant network is an ordinary tenant VXLAN network on the EVPN fabric; leg 2 is one more VM in it.
- In the portal: support enters the VLAN number (
l2_vlans); the tenant binds one of its VNets (shown as “name (id)”) to the VLAN without approval (l2_vnets), which reconfigures the legs once provisioning exists.
8. EVPN fabric (tenant overlays)¶
- Data plane — every tenant subnet is one VXLAN with a unique VNI. On a host, VMs of one VXLAN share a Linux bridge created with the first VM of that network on the host and removed after the last one. Between hosts, frames are carried over the isolated VXLAN transport segment, which has no L3 path outside.
- Control plane — BGP EVPN, FRR on every payload host. Hosts peer only with three route reflectors (VyOS VMs on the management platform) which distribute every VM's MAC/IP to all hosts; adding or removing a host changes nothing elsewhere.
- OpenNebula — the VTEP of every VXLAN network must be the host's local IP (
VXLAN_TEP = local_ip), on every VNet, on the VNet template tenants create networks from, and in the VXLAN driver of the FE nodes.
9. Addresses: pools, service networks, allocation¶
- Delegated subnets are the
bgp.subnetsof the exchange networks in the inventory. Public and corporate subnets are always bound to an IX network, and each IX network serves exactly one shared front end; front ends never share addresses or hosts. - Typically each site has a public pool on its cloud-ix and a private pool on its corpnet-ix; a secondary zone uses the pools of its own site. portal-ix pools belong to the portal platforms and are never used for tenants.
- Service networks: inventory networks inside a public pool (for example
cloud-services, the first /29 of each cloud-ix pool) carry the services on the management clusters. The portal records them in the address registry as platform services after every inventory change and never gives them to tenants. - Allocation: customers choose prefix sizes (/31, /29, /28, /27, /26); on the resource manager's approval the portal takes the first free aligned block, largest first, from the IX networks of the platform the tenant is placed on — cloud-ix for internet, corpnet-ix for the corporate network. Everything in the registry, in the latest old-system import and in OpenNebula group metadata counts as taken.
- The address registry lists pools, assignments (main, additional, corporate, platform services, reserved) and flags overlaps or assignments outside every pool.
10. Open points¶
- Corporate network: address plan vs tenant subnets (overlaps, NAT), one EGW for both exchange networks or one per network, route policy, bandwidth and filtering on the corporate EGW.
- L2 channels: MTU of the extended segment (VXLAN adds 50 bytes on the s2s transport), redundancy of the legs, image of the legs.
- Bandwidth step: the reference design shapes in 10 Mbit/s steps; the order form currently accepts 1 Mbit/s steps.