Proposed configuration for multi-tenant VAN participation

[ted-ross]

This discussion is about how workloads in a cluster are associated with specific VANs in a multi-tenant environment.

Some terms:

• Skupper Endpoint - A Listener, MultiKeyListener, or Connector in a namespace
• Network - A backbone interior network that carries multi-tenant VAN service traffic
• MultiTenantSite (MTSite) - A multi-tenant site that is configured on a cluster and is uplinked to a network

Skupper Endpoints

Skupper endpoint CRDs shall be enhanced by adding a spec.van attribute. This attribute is optional. Its type is an object with two fields:

• name -- The name of the VAN to which this endpoint is to connect.
• network -- (optional) name of the network on which the VAN is carried.

In determining how an endpoint is to be configured, the controller follows this algorithm:

if spec.van is present then
    spec.van.name is the multi-tenant VAN to be configured.
    if spec.van.network is present then
        spec.van.network is the network on which the VAN is assumed to be provisioned
    else
        Choose the network associated with the MTSite that is configured as "default" for the cluster
else
    if a Site CR (dedicated/legacy) is present in the namespace then
        Configure the endpoint on the local dedicated site
    else if a VAN CR is present in the namespace then
        Use the VAN-name and VAN-network specified in the VAN CR
    else
        Set the status of the endpoint CR as being in error state (no available VAN)

The VAN CRD

VAN CRs may optionally be configured in a namespace. The VAN CR identifies the default multi-tenant VAN for endpoints in the namespace.

If there are multiple VAN CRs in a namespace, only one is set as active in the same way that multiple Site CRs are handled.

If there are no VAN CRs in a namespace, all multi-tenant endpoints must explicitly identify a VAN(+Network)

Discussion

• The above allows a namespace to participate in multiple multi-tenant VANS.
• A namespace may have a dedicated (legacy) site and also contain endpoints that use multi-tenant VANs.
• The VAN name is constrained to be a DNS-name-segment (like a namespace name) and is used as the prefix for the routing key(s) configured for the endpoint. A Connector with VAN engteam and routing key backend will appear in the router as a connector for address engteam/backend.

...coming: Further discussion about access control for VANs and individual services and the trust-chain in the presence or absence of a centralized management function (VMS).

[fgiorgetti]

The proposed configuration makes sense to me.

One question related to .spec.van.network.
How are we going to map the provided value with an actual network?
It is related to a MTSite's field?

[ted-ross]

Yes, I think so. An MTSite will contain a field that identifies the name of the network to which it is connected. It will also have a boolean field to say whether this network is the cluster default.

The 99% case will involve a single MTSite in a cluster which will be the default network for all VAN endpoints.

The objective is to make it very easy to do the common things while still possible to do more advanced things like have a cluster that's connected to multiple networks.

[c-kruse]

I think that all makes sense to me.

I do get a smell from the "controller elects one winning resource" pattern: Site in a namespace, VAN CR in a namespace, MTSite with a network specified, MTSite with default set true. I think that strategy is full of sharp edges for us and for users. Thinking on whether there's a way to organize the API in way that makes the common paths just as easy without leaving it as open for misinterpretation.

If I had a suggestion so far it'd be some combination of ideas like this: forgoing the "network" identifier concept for a fully qualified MTSite reference, and using a special "default" name for resources intended as defaults.

In place of a VAN CR we have SkupperAttachment and ClusterSkupperAttachment resources with spec fields defaultVAN type string, mtSiteRef type object with name and namespace strings.

Endpoints keep .spec.van.name, but trade network for .spec.van.attachmentRef (string).

if spec.van is present then
    spec.van.name is the multi-tenant VAN to be configured.
    if spec.van.attachmentRef is set
        // resolve a MTSite from the SkupperAttachment (preferred) or ClusterSkupperAttachment with a matching name
    else if a SkupperAttachment exists with `.metadata.name == "default"`
        // resolve a MTSite from the SkupperAttachment named "default"
    else if a ClusterSkupperAttachment exists  with `.metadata.name == "default"`
        // resolve a MTSite from the ClusterSkupperAttachment named "default"
     else
        error - ambiguous attachment
else
    if a Site CR (dedicated/legacy) is present in the namespace then
        Configure the endpoint on the local dedicated site
    else if a SkupperAttachment exists with `.metadata.name == "default"`
        // resolve a MTSite AND VAN from the SkupperAttachment named "default"
    else
        Set the status of the endpoint CR as being in error state (no available VAN)

So our most common use case would be a ClusterSkupperAttachment named "default" referencing some MTSite. Endpoints need only set van.name or use the default.

[ted-ross]

I think the idea of a default VAN for a ClusterSkupperAttachment is a new and separate thing. It's not clear to me that it's useful for all namespaces in the cluster to have the same default VAN.

This also creates an ambiguity for the default VAN for endpoints if there's a dedicated site in the namespace.

i.e. If an endpoint does not specify a VAN and there is a dedicated Site in its namespace, does it default to the dedicated site or the default VAN?

[ted-ross]

Why is SkupperAttachment preferred over ClusterSkupperAttachment? I assume that the former is namespace scoped, in which case there can be multiples with the name default.

I didn't catch that you were considering SkupperAttachment to be a replacement to the VAN CR.

[ted-ross]

"Skupper" should probably not be used in a CRD name.

[c-kruse]

I think the idea of a default VAN for a ClusterSkupperAttachment is a new and separate thing. It's not clear to me that it's useful for all namespaces in the cluster to have the same default VAN.

Agreed - I had not thought about that option distinct from a namesapce-scoped SkupperAttachment indicating a default VAN for the namespace (which I do think is useful.) I could see us taking this a few ways: either omitting a defaultVAN spec field from the ClusterSkupperAttachment but leaving it on a namespace-scoped SkupperAttachment, OR pulling the concepts apart and having some hybrid of namespace-scoped VAN CRs and cluster scoped ClusterSkupperAttachment resources.

This also creates an ambiguity for the default VAN for endpoints if there's a dedicated site in the namespace.

I think that a dedicated Site should always win claim to endpoint resources without spec.van set (within the namespace, of course.)

*also not married to SkupperAttachment or any of my names, I just felt like I was drowning in VANs, so reached for a reasonable placeholder.

[ted-ross]

Ok, so the main things you are proposing are:

• Using the name default to explicitly identify the one object to use if not explicitly stated (rather than the controller choosing).
• Having a cluster-scoped CR that can identify the default MTSite to use if not explicitly stated.
• Reference named CRs rather than correlate on a "network name".

I think this is an improvement over my initial proposal.

We may feel like we're drowning in VANs, but we have never actually given a user something formal called "VAN". We may be able to get away with it.

[ted-ross]

Updated proposal based on the discussion thus far:

Changes

• Add a cluster-scoped CRD called something like NetworkAttach that references a namespaced MTSite. There is no reference to VAN in this CRD. The NetworkAttach with metadata.name: default is used where no NetworkAttach is specified.
• In the endpoints, spec.van now has a name and an optional NetworkAttach field.

Algorithm

if spec.van is present then
    spec.van.name is the multi-tenant VAN to be configured.
    if spec.van.networkAttach is present then
        spec.van.networkAttach references a cluster-scoped NetworkAttach that identifies the MTSite to be used
    else if there is a cluster-scoped NetworkAttach called `default` then
        The `default` NetworkAttach is referenced
    else
        Set the status of the endpoint CR as being in error state (no available NetworkAttach)
else
    if a Site CR (dedicated/legacy) is present in the namespace then
        Configure the endpoint on the local dedicated site
    else if a VAN CR with the name `default` is present in the namespace then
        Use the VAN-name and NetworkAttach (possibly defaulted) specified in the VAN CR
    else
        Set the status of the endpoint CR as being in error state (no available VAN)

[ted-ross]

note: based on the above, any VAN CR that is not named default will be ignored.