Tenants
Understand principles and concepts of Capsule Tenants
Capsule is a framework to implement multi-tenant and policy-driven scenarios in Kubernetes. In this tutorial, we’ll focus on a hypothetical case covering the main features of the Capsule Operator. This documentation is styled in a tutorial format, and it’s designed to be read in sequence. We’ll start with the basics and then move to more advanced topics.
Acme Corp, our sample organization, is building a Container as a Service platform (CaaS) to serve multiple lines of business, or departments, e.g. Oil, Gas, Solar, Wind, Water. Each department has its team of engineers that are responsible for the development, deployment, and operating of their digital products. We’ll work with the following actors:
- Bill: the cluster administrator from the operations department of Acme Corp.
- Alice: the project leader in the Solar & Green departments. She is responsible for a team made of different job responsibilities: e.g. developers, administrators, SRE engineers, etc.
- Joe: works as a lead developer of a distributed team in Alice’s organization.
- Bob: is the head of engineering for the Water department, the main and historical line of business at Acme Corp.
This scenario will guide you through the following topics.
1 - Quickstart
Create your first Capsule Tenant
In Capsule, a Tenant is an abstraction to group multiple namespaces in a single entity within a set of boundaries defined by the Cluster Administrator. The tenant is then assigned to a user or group of users who is called Tenant Owner. Capsule defines a Tenant as Custom Resource with cluster scope. Create the tenant as cluster admin:
kubectl create -f - << EOF
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
permissions:
matchOwners:
- matchLabels:
team: platform
owners:
- name: alice
kind: Us
You can check the tenant just created
```bash
$ kubectl get tenants
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR READY STATUS AGE
oil Active 0 True reconciled 13s
We create dedicated TenantOwners who represent cluster administrators. They are matched by labels defined in the permissions.matchOwners section of the Tenant spec. In our case, any user or group with the label team: platform is considered a TenantOwner for the oil tenant.
kubectl create -f - << EOF
apiVersion: capsule.clastix.io/v1beta2
kind: TenantOwner
metadata:
name: platform-team
labels:
team: platform
spec:
kind: Group
name: "oidc:kubernetes:admin"
EOF
We can now verify all owners of the oil tenant:
kubectl get tenant oil -o jsonpath='{.status.owners}'
The result should be similar to:
[
{
"kind": "Group",
"name": "oidc:kubernetes:admin",
"clusterRoles": [
"admin",
"capsule-namespace-deleter"
]
},
{
"kind": "User",
"name": "alice",
"clusterRoles": [
"admin",
"capsule-namespace-deleter"
]
}
]
Login as Tenant Owner
Each tenant comes with a delegated user or group of users acting as the tenant admin. In the Capsule jargon, this is called the Tenant Owners. Other users can operate inside a tenant with different levels of permissions and authorizations assigned directly by the Tenant Owner.
Capsule does not care about the authentication strategy used in the cluster and all the Kubernetes methods of authentication are supported. The only requirement to use Capsule is to assign tenant users to the group defined by –capsule-user-group option, which defaults to capsule.clastix.io.
Assignment to a group depends on the authentication strategy in your cluster.
For example, if you are using capsule.clastix.io, users authenticated through a X.509 certificate must have capsule.clastix.io as Organization: -subj "/CN=${USER}/O=capsule.clastix.io"
Users authenticated through an OIDC token must have in their token:
"users_groups": [
"projectcapsule.dev",
"other_group"
]
The hack/create-user.sh can help you set up a dummy kubeconfig for the alice user acting as owner of a tenant called solar.
./hack/create-user.sh alice solar
...
certificatesigningrequest.certificates.k8s.io/alice-solar created
certificatesigningrequest.certificates.k8s.io/alice-solar approved
kubeconfig file is: alice-solar.kubeconfig
to use it as alice export KUBECONFIG=alice-solar.kubeconfig
Login as tenant owner
Impersonation
You can simulate this behavior by using impersonation:
kubectl --as alice --as-group projectcapsule.dev ...
Create namespaces
As tenant owner, you can create namespaces:
kubectl create namespace solar-production
kubectl create namespace solar-development
or
kubectl --as alice --as-group projectcapsule.dev create namespace solar-production
kubectl --as alice --as-group projectcapsule.dev create namespace solar-development
And operate with fully admin permissions:
kubectl -n solar-development run nginx --image=docker.io/nginx
kubectl -n solar-development get pods
Limiting access
Tenant Owners have full administrative permissions limited to only the namespaces in the assigned tenant. They can create any namespaced resource in their namespaces but they do not have access to cluster resources or resources belonging to other tenants they do not own:
$ kubectl -n kube-system get pods
Error from server (Forbidden): pods is forbidden:
User "alice" cannot list resource "pods" in API group "" in the namespace "kube-system"
Securing The Network
As Cluster Administrator we want to avoid that tenants can communicate between each other. To achieve that, we can use Network Policies to isolate the namespaces created by different tenants.
Let’s ensure for any Tenant and any of its future namespaces, that each gets a NetworkPolicy, which does not allow ingress/egress traffic to/from other tenants.
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: default-networkpolicies
spec:
resyncPeriod: 60s
resources:
- rawItems:
- apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-policy
spec:
# Apply to all pods in this namespace
podSelector: {}
policyTypes:
- Ingress
- Egress
ingress:
# Allow traffic from the same namespace (intra-namespace communication)
- from:
- podSelector: {}
# Allow traffic from all namespaces within the tenant
- from:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: "{{tenant.name}}"
# Allow ingress from other namespaces labeled (System Namespaces, eg. Monitoring, Ingress)
- from:
- namespaceSelector:
matchLabels:
company.com/system: "true"
egress:
# Allow DNS to kube-dns service IP (might be different in your setup)
- to:
- ipBlock:
cidr: 10.96.0.10/32
ports:
- protocol: UDP
port: 53
- protocol: TCP
port: 53
# Allow traffic to all namespaces within the tenant
- to:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: "{{tenant.name}}"
2 - Namespaces
Namespace to Tenant relation
Alice, once logged with her credentials, can create a new Namespace in her Tenant, as simply issuing:
kubectl create ns solar-production
Alice started the name of the Namespace prepended by the name of the Tenant: this is not a strict requirement but it is highly suggested because it is likely that many different Tenants would like to call their Namespaces production, test, or demo, etc. The enforcement of this naming convention is optional and can be controlled by the cluster administrator with forceTenantPrefix option.
Alice can deploy any resource in any of the Namespaces. That is because she is the owner of the tenant solar and therefore she has full control over all Namespaces assigned to that Tenant.
kubectl -n solar-development run nginx --image=docker.io/nginx
kubectl -n solar-development get pods
Every Namespaces assigned to a Tenant has an owner reference pointing to the Tenant object itself. In Addition each Namespaces has a label capsule.clastix.io/tenant=<tenant_name> identifying the Tenant it belongs to (Read More).
The Namespaces are tracked as part of the Tenant status:
$ kubectl get tnt solar -o yaml
...
status:
...
# Simplie list of namespaces
namespaces:
- solar-dev
- solar-prod
- solar-test
# Size (Amount of namespaces)
size: 3
# Detailed information about each namespace
spaces:
- conditions:
- lastTransitionTime: "2025-12-04T10:23:17Z"
message: reconciled
reason: Succeeded
status: "True"
type: Ready
- lastTransitionTime: "2025-12-04T10:23:17Z"
message: not cordoned
reason: Active
status: "False"
type: Cordoned
metadata: {}
name: solar-prod
uid: ad8ea663-9457-4b00-ac67-0778c4160171
- conditions:
- lastTransitionTime: "2025-12-04T10:23:25Z"
message: reconciled
reason: Succeeded
status: "True"
type: Ready
- lastTransitionTime: "2025-12-04T10:23:25Z"
message: not cordoned
reason: Active
status: "False"
type: Cordoned
metadata: {}
name: solar-test
uid: 706e3d30-af2b-4acc-9929-acae7b887ab9
- conditions:
- lastTransitionTime: "2025-12-04T10:23:33Z"
message: reconciled
reason: Succeeded
status: "True"
type: Ready
- lastTransitionTime: "2025-12-04T10:23:33Z"
message: not cordoned
reason: Active
status: "False"
type: Cordoned
metadata: {}
name: solar-dev
uid: e4af5283-aad8-43ef-b8b8-abe7092e25d0
By default the following rules apply for namespaces:
- A
Namespace can not be moved from a Tenant to another one (or anywhere else). Namespaces are deleted when the Tenant is deleted.
If you feel like these rules are too restrictive, you must implement your own custom logic to handle these cases, for example, with Finalizers for Namespaces.
If namespaces are not correctly assigned to tenants, make sure to evaluate your Capsule Users Configuration.
Multiple Tenants
A single team is likely responsible for multiple lines of business. For example, in our sample organization Acme Corp., Alice is responsible for both the Solar and Green lines of business. It’s more likely that Alice requires two different Tenants, for example, solar and green to keep things isolated.
By design, the Capsule operator does not permit a hierarchy of Tenants, since all Tenants are at the same levels. However, we can assign the ownership of multiple Tenants to the same user or group of users.
Bill, the cluster admin, creates multiple Tenants having alice as owner:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
and
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: green
spec:
owners:
- name: alice
kind: User
Alternatively, the ownership can be assigned to a group called solar-and-green for both Tenants:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: solar-and-green
kind: Group
See Ownership for more details on how to assign ownership to a group of users.
The two tenants remain isolated from each other in terms of resources assignments, e.g. ResourceQuotas, Nodes, StorageClasses and IngressClasses, and in terms of governance, e.g. NetworkPolicies, PodSecurityPolicies, Trusted Registries, etc.
When Alice logs in, she has access to all namespaces belonging to both the solar and green Tenants.
Tenant Prefix
We recommend to use the forceTenantPrefix for production environments.
If the forceTenantPrefix option is enabled, which is not the case by default, the Namespaces are automatically assigned to the right tenant by Capsule because the operator does a lookup on the tenant names.
For example, Alice creates a Namespace called solar-production and green-production:
kubectl create ns solar-production
kubectl create ns green-production
And they are assigned to the Tenant based on their prefix:
$ kubectl get tnt
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
green Active 1 3m26s
solar Active 1 3m26s
However alice can create any Namespace, which does not have a prefix of any of the Tenants she owns, for example production:
$ kubectl create ns production
Error from server (Forbidden): admission webhook "owner.namespace.capsule.clastix.io" denied the request: The Namespace prefix used doesn't match any available Tenant
Label
The default behavior, if the forceTenantPrefix option is not enabled, Alice needs to specify the Tenant name as a label capsule.clastix.io/tenant=<desired_tenant> in the Namespace manifest:
kind: Namespace
apiVersion: v1
metadata:
name: solar-production
labels:
capsule.clastix.io/tenant: solar
If not specified, Capsule will deny with the following message: Unable to assign Namespace to Tenant:
$ kubectl create ns solar-production
Error from server (Forbidden): admission webhook "owner.namespace.capsule.clastix.io" denied the request: Please use capsule.clastix.io/tenant label when creating a namespace
Termination
Capsule keeps it’s managed resources as long as possible. Meaning even if a Namespace is terminated we verify the following things, before any capsule managed resources are finally removed:
- There are no pods in the namespace left (including pods with finalizers). Users must take own action to resolve finalizing pods.
- All other namespaced resources have removed finalizers and are deleted (capsule will initiate that step)
- Finally capsule managed resources are removed.
This ensures proper cleanup of namespaces without having namespaces being stuck in Terminating phase and requiring administrator attention.
If you are running a Kubernetes Version below 1.33 you should make sure, that the FeatureGate OrderedNamespaceDeletion is enabled. This already enforces this order by default:
- All Pods must be absent
- Only after all pods are absent other namespaced items are removed.
As stated in the following KEP.
Cordon
It is possible to cordon a Namespace from a Tenant, preventing anything from being changed within this Namespace. This is useful for production Namespaces where you want to avoid any accidental changes or if you have some sort of change freeze period.
This action can be performed by the TenantOwner by adding the label projectcapsule.dev/cordoned=true to the Namespace:
kubectl patch namespace solar-production --patch '{"metadata": {"labels": {"projectcapsule.dev/cordoned": "true"}}}' --as alice --as-group projectcapsule.dev
To uncordon the Namespace, simply remove the label or set it to false:
kubectl patch namespace solar-production --patch '{"metadata": {"labels": {"projectcapsule.dev/cordoned": "false"}}}' --as alice --as-group projectcapsule.dev
Note: If the entire Tenant is cordoned all Namespaces within the Tenant will be cordoned as well. Meaning a single Namespace can not be uncordoned if the Tenant is cordoned.
3 - Permissions
Grant permissions for tenants
Administrators
Administrators are users that have full control over all Tenants and their namespaces. They are typically cluster administrators or operators who need to manage the entire cluster and all its Tenants. However as administrator you are automatically Owner of all Tenants.Tenants This means that administrators can create, delete, and manage namespaces and other resources within any Tenant, given you are using label assignments for tenants.
Ownership
Capsule introduces the principal, that tenants must have owners (Tenant Owners). The owner of a tenant is a user or a group of users that have the right to create, delete, and manage the tenant’s namespaces and other tenant resources. However an owner does not have the permissions to manage the tenants they are owner of. This is still done by cluster-administrators.
At any time you are able to verify which users or groups are owners of a tenant by checking the owners field of the Tenant status subresource:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
...
status:
owners:
- clusterRoles:
- admin
- capsule-namespace-deleter
kind: Group
name: oidc:org:devops:a
- clusterRoles:
- admin
- capsule-namespace-deleter
- mega-admin
- controller
kind: ServiceAccount
name: system:serviceaccount:capsule:controller
- clusterRoles:
- admin
- capsule-namespace-deleter
kind: User
name: alice
To explain these entries, let’s inspect one of them:
kind: It can be User, Group or ServiceAccountname: Is the reference name of the user, group or serviceaccount we want to bindclusterRoles: ClusterRoles which are bound for each namespace of the tenant to the owner. By default, Capsule assigns admin and capsule-namespace-deleter roles to each owner, but you can customize them as explained in Owner Roles section.
With this information available you
Tenant Owners
Tenant Owners can be declared as dedicated cluster scoped Resources called TenantOwner. This allows the cluster admin to manage the ownership of tenants in a more flexible way, for example by adding labels and annotations to the TenantOwner resources.
apiVersion: capsule.clastix.io/v1beta2
kind: TenantOwner
metadata:
labels:
team: devops
name: devops
spec:
kind: Group
name: "oidc:org:devops:a"
This TenantOwner can now be matched by any tenant. Essentially we define on a per tenant basis which TenantOwners should be owners of the tenant (Each item under spec.permissions.matchOwners is understood as OR selection.):
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
labels:
kubernetes.io/metadata.name: solar
name: solar
spec:
permissions:
matchOwners:
- matchLabels:
team: devops
- matchLabels:
customer: x
Since the ownership is now loosely coupled, all TenantOwners matching the given labels will be owners of the tenant. We can verify this via the .status.owners field of the Tenant resource:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
...
status:
owners:
- clusterRoles:
- admin
- capsule-namespace-deleter
kind: Group
name: oidc:org:devops:a
- clusterRoles:
- admin
- capsule-namespace-deleter
kind: User
name: alice
This can also be combined with direct owner declarations. In the example, both alice user and all TenantOwners with label team: devops and TenantOwners with label customer: x will be owners of the solar tenant.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- clusterRoles:
- admin
- capsule-namespace-deleter
kind: User
name: alice
- clusterRoles:
- admin
- capsule-namespace-deleter
kind: ServiceAccount
name: system:serviceaccount:capsule:controller
permissions:
matchOwners:
- matchLabels:
team: devops
- matchLabels:
customer: x
If we create a TenantOwner where the .spec.name and .spec.kind matches one of the owners declared in the tenant, the entries wille be merged. That’s mainly relevant for the clusterRoles:
apiVersion: capsule.clastix.io/v1beta2
kind: TenantOwner
metadata:
labels:
customer: x
name: controller
spec:
kind: ServiceAccount
name: "system:serviceaccount:capsule:controller"
clusterRoles:
- "mega-admin"
- "controller"
Again we can verify the resulting owners via the .status.owners field of the Tenant resource:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
...
status:
owners:
- clusterRoles:
- admin
- capsule-namespace-deleter
kind: Group
name: oidc:org:devops:a
- clusterRoles:
- admin
- capsule-namespace-deleter
- mega-admin
- controller
kind: ServiceAccount
name: system:serviceaccount:capsule:controller
- clusterRoles:
- admin
- capsule-namespace-deleter
kind: User
name: alice
We can see that the system:serviceaccount:capsule:controller ServiceAccount now has additional mega-admin and controller roles assigned.
Implicit Tenant Assignment
If a TenantOwner is created all Tenants are always matching the label projectcapsule.dev/tenant on TenantOwner with the name of the Tenant. This means that if you create a TenantOwner with the name solar, it will automatically become owner of the solar Tenant. This can only be done for one tenant at a time (because the label is unique) and is intended that way.
apiVersion: capsule.clastix.io/v1beta2
kind: TenantOwner
metadata:
labels:
projectcapsule.dev/tenant: "solar"
name: solar-test-gitops-reconciler
spec:
kind: ServiceAccount
name: "system:serviceaccount:solar-test:gitops-reconciler"
With this Tenant specification:
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec: {}
We can observe that the owner is automatically assigned for the Tenant solar:
kubectl get tnt solar -o yaml
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
cordoned: false
preventDeletion: false
status:
owners:
- clusterRoles:
- admin
- capsule-namespace-deleter
kind: ServiceAccount
name: system:serviceaccount:solar-test:gitops-reconciler
Aggregation
All subjects defined in TenantOwner resources are automatically considered Capsule Users and don’t need to mentioned further in the CapsuleConfiguration User Scope. If you don’t want this behavior, you can disable it by setting aggregate: false in the TenantOwner spec:
apiVersion: capsule.clastix.io/v1beta2
kind: TenantOwner
metadata:
labels:
customer: x
name: controller
spec:
kind: ServiceAccount
name: "system:serviceaccount:capsule:controller"
aggregate: false
Users
Bill, the cluster admin, receives a new request from Acme Corp’s CTO asking for a new Tenant to be onboarded and Alice user will be the TenantOwner. Bill then assigns Alice’s identity of alice in the Acme Corp. identity management system. Since Alice is a TenantOwner, Bill needs to assign alice the Capsule group defined by –capsule-user-group option, which defaults to projectcapsule.dev.
To keep things simple, we assume that Bill just creates a client certificate for authentication using X.509 Certificate Signing Request, so Alice’s certificate has "/CN=alice/O=projectcapsule.dev".
Bill creates a new Tenant solar in the CaaS management portal according to the Tenant’s profile:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
Bill checks if the new Tenant is created and operational:
kubectl get tenant solar
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
solar Active 0 33m
Note that namespaces are not yet assigned to the new Tenant. The Tenant owners are free to create their namespaces in a self-service fashion and without any intervention from Bill.
Once the new Tenant solar is in place, Bill sends the login credentials to Alice. Alice can log in using her credentials and check if she can create a namespace
kubectl auth can-i create namespaces
yes
or even delete the namespace
kubectl auth can-i delete ns -n solar-production
yes
However, cluster resources are not accessible to Alice
kubectl auth can-i get namespaces
no
kubectl auth can-i get nodes
no
kubectl auth can-i get persistentvolumes
no
including the Tenant resources
kubectl auth can-i get tenants
no
Groups
In the example above, Bill assigned the ownership of solar Tenant to alice user. If another user, e.g. Bob needs to administer the solar Tenant, Bill can assign the ownership of solar Tenant to such user too:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
- name: bob
kind: User
However, it’s more likely that Bill assigns the ownership of the solar Tenant to a group of users instead of a single one, especially if you use OIDC Authentication. Bill creates a new group account solar-users in the Acme Corp. identity management system and then he assigns Alice and Bob identities to the solar-users group.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: solar-users
kind: Group
With the configuration above, any user belonging to the solar-users group will be the owner of the solar Tenant with the same permissions of Alice. For example, Bob can log in with his credentials and issue
kubectl auth can-i create namespaces
yes
All the groups you want to promote to TenantOwners must be part of the Group Scope. You have to add solar-users to the CapsuleConfiguration Group Scope to make it work.
ServiceAccounts
You can use the Group subject to grant ServiceAccounts the ownership of a Tenant. For example, you can create a group of ServiceAccounts and assign it to the Tenant:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: system:serviceaccount:tenant-system:robot
kind: ServiceAccount
Bill can create a ServiceAccount called robot, for example, in the tenant-system namespace and leave it to act as TenantOwner of the solar Tenant
kubectl --as system:serviceaccount:tenant-system:robot --as-group projectcapsule.dev auth can-i create namespaces
yes
since each service account in a namespace is a member of following group:
system:serviceaccounts:{service-account-namespace}
You have to add system:serviceaccounts:{service-account-namespace} to the CapsuleConfiguration Group Scope or system:serviceaccounts:{service-account-namespace}:{service-account-name} to the CapsuleConfiguration User Scope to make it work.
Owner Roles
By default, all TenantOwners will be granted with two ClusterRole resources using the RoleBinding API:
admin: the Kubernetes default one, admin, that grants most of the namespace scoped resourcescapsule-namespace-deleter: a custom clusterrole, created by Capsule, allowing to delete the created namespaces
You can observe this behavior when you get the Tenant solar:
$ kubectl get tnt solar -o yaml
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
labels:
kubernetes.io/metadata.name: solar
name: solar
spec:
ingressOptions:
hostnameCollisionScope: Disabled
limitRanges: {}
networkPolicies: {}
owners:
# -- HERE -- #
- clusterRoles:
- admin
- capsule-namespace-deleter
kind: User
name: alice
labels:
projectcapsule.dev/sample: "true"
annotations:
projectcapsule.dev/sample: "true"
resourceQuotas:
scope: Tenant
status:
namespaces:
- solar-production
- solar-system
size: 2
state: Active
In the example below, assuming the TenantOwner creates a namespace solar-production in Tenant solar, you’ll see the Role Bindings giving the TenantOwner full permissions on the Tenant namespaces:
$ kubectl get rolebinding -n solar-production
NAME ROLE AGE
capsule-solar-0-admin ClusterRole/admin 111m
capsule-solar-1-capsule-namespace-deleter ClusterRole/capsule-namespace-deleter 111m
When Alice creates the namespaces, the Capsule controller assigns to Alice the following permissions, so that Alice can act as the admin of all the Tenant namespaces:
$ kubectl get rolebinding -n solar-production -o yaml
apiVersion: v1
items:
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: "2024-02-25T14:02:36Z"
labels:
capsule.clastix.io/role-binding: 8fb969aaa7a67b71
capsule.clastix.io/tenant: solar
projectcapsule.dev/sample: "true"
annotations:
projectcapsule.dev/sample: "true"
name: capsule-solar-0-admin
namespace: solar-production
ownerReferences:
- apiVersion: capsule.clastix.io/v1beta2
blockOwnerDeletion: true
controller: true
kind: Tenant
name: solar
uid: 1e6f11b9-960b-4fdd-82ee-7cd91a2db052
resourceVersion: "2980"
uid: 939da5ae-7fec-4300-8db2-223d3049b43f
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: admin
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: alice
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
creationTimestamp: "2024-02-25T14:02:36Z"
labels:
capsule.clastix.io/role-binding: b8822dde20953fb1
capsule.clastix.io/tenant: solar
projectcapsule.dev/sample: "true"
annotations:
projectcapsule.dev/sample: "true"
name: capsule-solar-1-capsule-namespace-deleter
namespace: solar-production
ownerReferences:
- apiVersion: capsule.clastix.io/v1beta2
blockOwnerDeletion: true
controller: true
kind: Tenant
name: solar
uid: 1e6f11b9-960b-4fdd-82ee-7cd91a2db052
resourceVersion: "2982"
uid: bbb4cd79-ce0d-41b0-a52d-dbed71a9b48a
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: capsule-namespace-deleter
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: alice
kind: List
metadata:
resourceVersion: ""
In some cases, the cluster admin needs to narrow the range of permissions assigned to TenantOwners by assigning a Cluster Role with less permissions than above. Capsule supports the dynamic assignment of any ClusterRole resources for each TenantOwner.
For example, assign user Joe the Tenant ownership with only view permissions on Tenant namespaces:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
- name: joe
kind: User
clusterRoles:
- view
you’ll see the new Role Bindings assigned to Joe:
$ kubectl get rolebinding -n solar-production
NAME ROLE AGE
capsule-solar-0-admin ClusterRole/admin 114m
capsule-solar-1-capsule-namespace-deleter ClusterRole/capsule-namespace-deleter 114m
capsule-solar-2-view ClusterRole/view 1s
so that Joe can only view resources in the Tenant namespaces:
kubectl --as joe --as-group projectcapsule.dev auth can-i delete pods -n solar-production
no
Please, note that, despite created with more restricted permissions, a TenantOwner can still create namespaces in the Tenant because he belongs to the projectcapsule.dev group. If you want a user not acting as TenantOwner, but still operating in the Tenant, you can assign additional RoleBindings without assigning him the Tenant ownership.
Custom ClusterRoles are also supported. Assuming the cluster admin creates:
kubectl apply -f - << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tenant-resources
rules:
- apiGroups: ["capsule.clastix.io"]
resources: ["tenantresources"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
EOF
These permissions can be granted to Joe
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
- name: joe
kind: User
clusterRoles:
- view
- tenant-resources
For the given configuration, the resulting RoleBinding resources are the following ones:
$ kubectl -n solar-production get rolebindings
NAME ROLE AGE
capsule-solar-0-admin ClusterRole/admin 90s
capsule-solar-1-capsule-namespace-deleter ClusterRole/capsule-namespace-deleter 90s
capsule-solar-2-view ClusterRole/view 90s
capsule-solar-3-tenant-resources ClusterRole/prometheus-servicemonitors-viewer 25s
Role Aggregation
Sometimes the admin role is missing certain permissions. You can aggregate the admin role with a custom role, for example, gateway-resources:
kubectl apply -f - << EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: gateway-resources
labels:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["gateway.networking.k8s.io"]
resources: ["gateways"]
verbs: ["*"]
EOF
Proxy Owner Authorization
This feature will be deprecated in a future release of Capsule. Instead use ProxySettings
When you are using the Capsule Proxy, the tenant owner can list the cluster-scoped resources. You can control the permissions to cluster scoped resources by defining proxySettings for a tenant owner.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: joe
kind: User
clusterRoles:
- view
- tenant-resources
As Tenant Owner you can perform ServiceAccount Promotion.
Within a Tenant, a ServiceAccount can be promoted to a TenantOwner. For example, Alice can create a ServiceAccount called robot in the solar Tenant and promote it to be a TenantOwner (This requires Alice to be an owner of the Tenant as well):
kubectl label sa gitops-reconcile -n green-test owner.projectcapsule.dev/promote=true --as alice --as-group projectcapsule.dev
Note: Promotion is only triggered on the label owner.projectcapsule.dev/promote with the value true
We can now verify if the promotion was successful by checking the Tenant status:
kubectl get tnt green -o jsonpath='{.status.owners}' | jq
[
{
"clusterRoles": [
"capsule-namespace-provisioner",
"capsule-namespace-deleter"
],
"kind": "ServiceAccount",
"name": "system:serviceaccount:green-test:gitops-reconcile"
},
{
"clusterRoles": [
"view",
"tenant-resources"
],
"kind": "User",
"name": "joe"
}
]
Now the ServiceAccount robot can create namespaces in the solar Tenant:
kubectl create ns green-valkey--as system:serviceaccount:green-test:gitops-reconcile
To revoke the promotion, Alice can just remove the label:
kubectl label sa gitops-reconcile -n green-test owner.projectcapsule.dev/promote- --as alice --as-group projectcapsule.dev
This feature must be enabled in the CapsuleConfiguration. The ClusterRoles assigned to promoted ServiceAccounts can be configured in the CapsuleConfiguration as well.
You can also dis/enable Owner Promotion per Tenant. By default it’s enabled, however since it’s disabled in the CapsuleConfiguration it can’t be used, unless that’s enabled as well.
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
permissions:
promotions:
allowOwnerPromotion: false
Read More
Additional Rolebindings
With Tenant rolebindings you can distribute namespaced rolebindings to all namespaces which are assigned to a namespace. Essentially it is then ensured the defined rolebindings are present and reconciled in all namespaces of the Tenant. This is useful if users should have more insights on Tenant basis. Let’s look at an example.
Assuming a cluster-administrator creates the following clusterRole:
kubectl apply -f - << EOF
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: prometheus-servicemonitors-viewer
rules:
- apiGroups: ["monitoring.coreos.com"]
resources: ["servicemonitors"]
verbs: ["get", "list", "watch"]
EOF
Now the cluster-administrator creates wants to bind this clusterRole in each namespace of the solar Tenant. He creates a tenantRoleBinding:
kubectl apply -f - << EOF
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
additionalRoleBindings:
- clusterRoleName: 'prometheus-servicemonitors-viewer'
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: joe
labels:
projectcapsule.dev/sample: "true"
annotations:
projectcapsule.dev/sample: "true"
EOF
As you can see the subjects is a classic rolebinding subject. This way you grant permissions to the subject user Joe, who only can list and watch servicemonitors in the solar tenant namespaces, but has no other permissions.
Strict
If you have strict RBAC enabled for the controller, you need to ensure that the controller ServiceAccount has the permission to create RoleBindings for the specified ClusterRole. The Controller Aggregates ClusterRoles with the labels (OR):
projectcapsule.dev/aggregate-to-controller: "true"projectcapsule.dev/aggregate-to-controller-instance: {{ .Release.Name }}
So for the above example, you need to label the prometheus-servicemonitors-viewer ClusterRole like this:
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: prometheus-servicemonitors-viewer
labels:
projectcapsule.dev/aggregate-to-controller: "true"
rules:
- apiGroups: ["monitoring.coreos.com"]
resources: ["servicemonitors"]
verbs: ["get", "list", "watch"]
Built-in ClusterRoles
We strongly recommend you use custom ClusterRoles for your Tenant rolebindings, but you can also use built-in ClusterRoles (admin (default for Tenant Owners), view and edit). For example, if you want to give the view permissions to Joe in all namespaces of the solar Tenant, you can use the built-in view ClusterRole.
In that case it also makes sense to use ClusterRole Aggregation. In the following example we are creating custom aggregated ClusterRoles for these three built-in clusterroles, to allow interactions with the GatewayAPI resources:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tenant:admins:extension
labels:
rbac.authorization.k8s.io/aggregate-to-admin: "true"
rules:
- apiGroups: ["gateway.networking.k8s.io"]
resources:
- gateways
- httproutes
- grpcroutes
- tlsroutes
- tcproutes
- udproutes
- referencegrants
- backendtlspolicies
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["gateway.networking.k8s.io"]
resources:
- gateways/status
- httproutes/status
- grpcroutes/status
- tlsroutes/status
- tcproutes/status
- udproutes/status
- referencegrants/status
- backendtlspolicies/status
verbs: ["get"]
- apiGroups: ["gateway.envoyproxy.io"]
resources:
- clienttrafficpolicies
- backendtrafficpolicies
- securitypolicies
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["gateway.envoyproxy.io"]
resources:
- clienttrafficpolicies/status
- backendtrafficpolicies/status
- securitypolicies/status
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tenant:members:extension
labels:
rbac.authorization.k8s.io/aggregate-to-edit: "true"
rules:
- apiGroups: ["gateway.networking.k8s.io"]
resources:
- gateways
- httproutes
- grpcroutes
- tlsroutes
- tcproutes
- udproutes
- referencegrants
- backendtlspolicies
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["gateway.networking.k8s.io"]
resources:
- gateways/status
- httproutes/status
- grpcroutes/status
- tlsroutes/status
- tcproutes/status
- udproutes/status
- referencegrants/status
- backendtlspolicies/status
verbs: ["get"]
- apiGroups: ["gateway.envoyproxy.io"]
resources:
- clienttrafficpolicies
- backendtrafficpolicies
- securitypolicies
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["gateway.envoyproxy.io"]
resources:
- clienttrafficpolicies/status
- backendtrafficpolicies/status
- securitypolicies/status
verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: tenant:viewers:extension
labels:
rbac.authorization.k8s.io/aggregate-to-view: "true"
rules:
- apiGroups: ["gateway.networking.k8s.io"]
resources:
- gateways
- httproutes
- grpcroutes
- tlsroutes
- tcproutes
- udproutes
- referencegrants
- backendtlspolicies
verbs: ["get", "list", "watch"]
- apiGroups: ["gateway.networking.k8s.io"]
resources:
- gateways/status
- httproutes/status
- grpcroutes/status
- tlsroutes/status
- tcproutes/status
- udproutes/status
- referencegrants/status
- backendtlspolicies/status
verbs: ["get"]
- apiGroups: ["gateway.envoyproxy.io"]
resources:
- clienttrafficpolicies
- backendtrafficpolicies
- securitypolicies
verbs: ["get", "list", "watch", "create"]
- apiGroups: ["gateway.envoyproxy.io"]
resources:
- clienttrafficpolicies/status
- backendtrafficpolicies/status
- securitypolicies/status
verbs: ["get"]
Selective Distribution
You may have the use-case where you want to distribute different ClusterRoles to different namespaces of the same Tenant. For example, you want to give view permissions to a operational group in all namespaces of the solar Tenant with environment=production label, but you want to give edit permissions to the operations group inall other namespaces. You can achieve this by leveraging GlobalTenantResources:
---
apiVersion: capsule.clastix.io/v1beta2
kind: GlobalTenantResource
metadata:
name: operators-rolebindings
spec:
resyncPeriod: 60s
resources:
- namespaceSelector:
matchExpressions:
- key: environment
operator: NotIn
values:
- prod
rawItems:
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: operators-rw
subjects:
- kind: Group
name: tenant:{{tenant.name}}:operators
namespace: "{{namespace}}"
roleRef:
kind: ClusterRole
name: view
apiGroup: rbac.authorization.k8s.io
- namespaceSelector:
matchLabels:
environment: prod
rawItems:
- apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: operators-view-only
subjects:
- kind: Group
name: tenant:{{tenant.name}}:operators
namespace: "{{namespace}}"
roleRef:
kind: ClusterRole
name: edit
apiGroup: rbac.authorization.k8s.io
Custom Resources
Capsule grants admin permissions to the TenantOwners but is only limited to their namespaces. To achieve that, it assigns the ClusterRole admin to the TenantOwner. This ClusterRole does not permit the installation of custom resources in the namespaces.
In order to leave the TenantOwner to create Custom Resources in their namespaces, the cluster admin defines a proper Cluster Role. For example:
kubectl apply -f - << EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: argoproj-provisioner
rules:
- apiGroups:
- argoproj.io
resources:
- applications
- appprojects
verbs:
- create
- get
- list
- watch
- update
- patch
- delete
EOF
Bill can assign this role to any namespace in the Alice’s Tenant by setting it in the Tenant manifest:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
- name: joe
kind: User
additionalRoleBindings:
- clusterRoleName: 'argoproj-provisioner'
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: alice
- apiGroup: rbac.authorization.k8s.io
kind: User
name: joe
With the given specification, Capsule will ensure that all Alice’s namespaces will contain a RoleBinding for the specified Cluster Role. For example, in the solar-production namespace, Alice will see:
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: capsule-solar-argoproj-provisioner
namespace: solar-production
subjects:
- kind: User
apiGroup: rbac.authorization.k8s.io
name: alice
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: argoproj-provisioner
With the above example, Capsule is leaving the TenantOwner to create namespaced custom resources.
Take Note: a TenantOwner having the admin scope on its namespaces only, does not have the permission to create Custom Resources Definitions (CRDs) because this requires a cluster admin permission level. Only Bill, the cluster admin, can create CRDs. This is a known limitation of any multi-tenancy environment based on a single shared control plane.
4 - Quotas
Strategies on granting quotas on tenant-basis
With help of Capsule, Bill, the cluster admin, can set and enforce resources quota and limits for Alice’s Tenant.
Resource Quota
Deprecated
This feature will be deprecated in a future release of Capsule. Instead use
Resource Pools or
Custom Quotas to handle any cases around distributed ResourceQuotas
With help of Capsule, Bill, the cluster admin, can set and enforce resources quota and limits for Alice’s Tenant. Set resources quota for each Namespace in the Alice’s Tenant by defining them in the Tenant spec:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
namespaceOptions:
quota: 3
resourceQuotas:
scope: Tenant
items:
- hard:
limits.cpu: "8"
limits.memory: 16Gi
requests.cpu: "8"
requests.memory: 16Gi
- hard:
pods: "10"
The resource quotas above will be inherited by all the Namespaces created by Alice. In our case, when Alice creates the Namespace solar-production, Capsule creates the following resource quotas:
kind: ResourceQuota
apiVersion: v1
metadata:
name: capsule-solar-0
namespace: solar-production
labels:
tenant: solar
spec:
hard:
limits.cpu: "8"
limits.memory: 16Gi
requests.cpu: "8"
requests.memory: 16Gi
---
kind: ResourceQuota
apiVersion: v1
metadata:
name: capsule-oil-1
namespace: solar-production
labels:
tenant: solar
spec:
hard:
pods : "10"
Alice can create any resource according to the assigned quotas:
kubectl -n solar-production create deployment nginx --image nginx:latest --replicas 4
At Namespaces solar-production level, Alice can see the used resources by inspecting the status in ResourceQuota:
kubectl -n solar-production get resourcequota capsule-solar-1 -o yaml
...
status:
hard:
pods: "10"
services: "50"
used:
pods: "4"
When defining ResourceQuotas you might want to consider distributing LimitRanges via Tenant Replications:
apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
name: solar-limitranges
namespace: solar-system
spec:
resyncPeriod: 60s
resources:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: solar
rawItems:
- apiVersion: v1
kind: LimitRange
metadata:
name: cpu-resource-constraint
spec:
limits:
- default: # this section defines default limits
cpu: 500m
defaultRequest: # this section defines default requests
cpu: 500m
max: # max and min define the limit range
cpu: "1"
min:
cpu: 100m
type: Container
Tenant Scope
By setting enforcement at Tenant level, i.e. spec.resourceQuotas.scope=Tenant, Capsule aggregates resources usage for all Namespaces in the Tenant and adjusts all the ResourceQuota usage as aggregate. In such case, Alice can check the used resources at the Tenant level by inspecting the annotations in ResourceQuota object of any Namespace in the Tenant:
kubectl -n solar-production get resourcequotas capsule-solar-1 -o yaml
apiVersion: v1
kind: ResourceQuota
metadata:
annotations:
quota.capsule.clastix.io/used-pods: "4"
quota.capsule.clastix.io/hard-pods: "10"
...
or
kubectl -n solar-development get resourcequotas capsule-solar-1 -o yaml
apiVersion: v1
kind: ResourceQuota
metadata:
annotations:
quota.capsule.clastix.io/used-pods: "4"
quota.capsule.clastix.io/hard-pods: "10"
...
When the aggregate usage for all Namespaces crosses the hard quota, then the native ResourceQuota Admission Controller in Kubernetes denies Alice’s request to create resources exceeding the quota:
kubectl -n solar-development create deployment nginx --image nginx:latest --replicas 10
Alice cannot schedule more pods than the admitted at Tenant aggregate level.
kubectl -n solar-development get pods
NAME READY STATUS RESTARTS AGE
nginx-55649fd747-6fzcx 1/1 Running 0 12s
nginx-55649fd747-7q6x6 1/1 Running 0 12s
nginx-55649fd747-86wr5 1/1 Running 0 12s
nginx-55649fd747-h6kbs 1/1 Running 0 12s
nginx-55649fd747-mlhlq 1/1 Running 0 12s
nginx-55649fd747-t48s5 1/1 Running 0 7s
and
kubectl -n solar-production get pods
NAME READY STATUS RESTARTS AGE
nginx-55649fd747-52fsq 1/1 Running 0 22m
nginx-55649fd747-9q8n5 1/1 Running 0 22m
nginx-55649fd747-r8vzr 1/1 Running 0 22m
nginx-55649fd747-tkv7m 1/1 Running 0 22m
Namespace Scope
By setting enforcement at the Namespace level, i.e. spec.resourceQuotas.scope=Namespace, Capsule does not aggregate the resources usage and all enforcement is done at the Namespace level.
Namespace Quotas
The cluster admin, can control how many Namespaces Alice, creates by setting a quota in the Tenant manifest spec.namespaceOptions.quota:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
namespaceOptions:
quota: 3
Alice can create additional Namespaces according to the quota:
kubectl create ns solar-development
kubectl create ns solar-test
While Alice creates Namespaces, the Capsule controller updates the status of the Tenant so Bill, the cluster admin, can check the status:
$ kubectl describe tenant solar
...
status:
Namespaces:
solar-development
solar-production
solar-test
Size: 3 # current namespace count
State: Active
...
Once the Namespace quota assigned to the tenant has been reached, Alice cannot create further Namespaces:
$ kubectl create ns solar-training
Error from server (Cannot exceed Namespace quota: please, reach out to the system administrators):
admission webhook "namespace.capsule.clastix.io" denied the request.
The enforcement on the maximum number of Namespaces per Tenant is the responsibility of the Capsule controller via its Dynamic Admission Webhook capability.
Custom Resources
This feature is still in an alpha stage and requires a high amount of computing resources due to the dynamic client requests.
Kubernetes offers by default ResourceQuota resources, aimed to limit the number of basic primitives in a Namespace.
Capsule already provides the sharing of these constraints across the Tenant Namespaces, however, limiting the amount of namespaced Custom Resources instances is not upstream-supported.
Starting from Capsule v0.1.1, this can be done using a special annotation in the Tenant manifest.
Imagine the case where a Custom Resource named mysqls in the API group databases.acme.corp/v1 usage must be limited in the Tenant solar: this can be done as follows.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
annotations:
quota.resources.capsule.clastix.io/mysqls.databases.acme.corp_v1: "3"
spec:
additionalRoleBindings:
- clusterRoleName: mysql-namespace-admin
subjects:
- kind: User
name: alice
owners:
- name: alice
kind: User
The Additional Role Binding referring to the Cluster Role mysql-namespace-admin is required to let Alice manage their Custom Resource instances.
The pattern for the quota.resources.capsule.clastix.io annotation is the following:
quota.resources.capsule.clastix.io/${PLURAL_NAME}.${API_GROUP}_${API_VERSION}
You can figure out the required fields using kubectl api-resources.
When alice will create a MySQL instance in one of their Tenant Namespace, the Cluster Administrator can easily retrieve the overall usage.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
annotations:
quota.resources.capsule.clastix.io/mysqls.databases.acme.corp_v1: "3"
used.resources.capsule.clastix.io/mysqls.databases.acme.corp_v1: "1"
spec:
owners:
- name: alice
kind: User
Node Pools
Bill, the cluster admin, can dedicate a pool of worker nodes to the oil Tenant, to isolate the Tenant applications from other noisy neighbors. To achieve this approach use NodeSelectors.
5 - Administration
Administrative controls on tenants
Cordoning
Bill needs to cordon a Tenant and its Namespaces for several reasons:
- Avoid accidental resource modification(s) including deletion during a Production Freeze Window
- During the Kubernetes upgrade, to prevent any workload updates
- During incidents or outages
- During planned maintenance of a dedicated nodes pool in a BYOD scenario
With the default installation of Capsule all CREATE, UPDATE and DELETE operations performed by Capsule Users are dropped. Any Updates to Subresources (i.e. status updates) and events are allowed to proceed as usual. If you wish to allow specific Operations, you can change the values for the Cordoning Admission via Values (eg. allow Pod/DELETE operations):
webhooks:
hooks:
cordoning:
matchConditions:
- name: skip-pod-create-delete
expression: '!(request.resource.resource == "pods" && request.operation in ["DELETE"])'
# Default conditions to ignore subresources and events
- name: ignore-subresources
expression: '!has(request.subResource) || request.subResource == ""'
- name: ignore-events
expression: 'request.resource.resource != "events"'
This is possible by just toggling the specific Tenant specification:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
cordoned: true
owners:
- kind: User
name: alice
Any operation performed by Alice, the TenantOwner, will be rejected by the Admission controller:
kubectl delete pod --all -n solar-test --as alice --as-group projectcapsule.dev
Error from server (Forbidden): admission webhook "cordoning.misc.projectcapsule.dev" denied the request: The current namespace 'solar-test' is cordoned. The attempted operation DELETE for /v1/Pod/nginx-deployment-56f567c7cb-pj86t is not permitted during cordoning status.
Uncordoning can be done by removing the said specification key:
$ cat <<EOF | kubectl apply -f -
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
cordoned: false
owners:
- kind: User
name: alice
EOF
$ kubectl --as alice --as-group projectcapsule.dev -n solar-dev create deployment nginx --image nginx
deployment.apps/nginx created
Status of cordoning is also reported in the state of the Tenant:
kubectl get tenants
NAME STATE NAMESPACE QUOTA NAMESPACE COUNT NODE SELECTOR AGE
bronze Active 2 3d13h
gold Active 2 3d13h
solar Cordoned 4 2d11h
silver Active 2 3d13h
Force Tenant-Prefix
Use this if you want to disable/enable the Tenant name prefix to specific Tenants, overriding global forceTenantPrefix in CapsuleConfiguration. When set to ’true’, it enforces Namespaces created for this Tenant to be named with the Tenant name prefix, separated by a dash (i.e. for Tenant ‘foo’, Namespace names must be prefixed with ‘foo-’), this is useful to avoid Namespace name collision. When set to ‘false’, it allows Namespaces created for this Tenant to be named anything. Overrides CapsuleConfiguration global forceTenantPrefix for the Tenant only. If unset, Tenant uses CapsuleConfiguration’s forceTenantPrefix
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
forceTenantPrefix: true
Deletion Protection
Sometimes it is important to protect business critical Tenants from accidental deletion. This can be achieved by toggling preventDeletion specification key on the Tenant:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
preventDeletion: true
6 - Enforcement
Configure policies and restrictions on tenant-basis
Scheduling
LimitRanges
This feature will be deprecated in a future release of Capsule. Instead use TenantReplications
Bill, the cluster admin, can also set Limit Ranges for each Namespace in Alice’s Tenant by defining limits for pods and containers in the Tenant spec:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
...
limitRanges:
items:
- limits:
- type: Pod
min:
cpu: "50m"
memory: "5Mi"
max:
cpu: "1"
memory: "1Gi"
- limits:
- type: Container
defaultRequest:
cpu: "100m"
memory: "10Mi"
default:
cpu: "200m"
memory: "100Mi"
min:
cpu: "50m"
memory: "5Mi"
max:
cpu: "1"
memory: "1Gi"
- limits:
- type: PersistentVolumeClaim
min:
storage: "1Gi"
max:
storage: "10Gi"
Limits will be inherited by all the Namespaces created by Alice. In our case, when Alice creates the Namespace solar-production, Capsule creates the following:
apiVersion: v1
kind: LimitRange
metadata:
name: capsule-solar-0
namespace: solar-production
spec:
limits:
- max:
cpu: "1"
memory: 1Gi
min:
cpu: 50m
memory: 5Mi
type: Pod
---
apiVersion: v1
kind: LimitRange
metadata:
name: capsule-solar-1
namespace: solar-production
spec:
limits:
- default:
cpu: 200m
memory: 100Mi
defaultRequest:
cpu: 100m
memory: 10Mi
max:
cpu: "1"
memory: 1Gi
min:
cpu: 50m
memory: 5Mi
type: Container
---
apiVersion: v1
kind: LimitRange
metadata:
name: capsule-solar-2
namespace: solar-production
spec:
limits:
- max:
storage: 10Gi
min:
storage: 1Gi
type: PersistentVolumeClaim
Note: being the limit range specific of single resources, there is no aggregate to count.
Alice doesn’t have permission to change or delete the resources according to the assigned RBAC profile.
kubectl -n solar-production auth can-i patch resourcequota
no
kubectl -n solar-production auth can-i delete resourcequota
no
kubectl -n solar-production auth can-i patch limitranges
no
kubectl -n solar-production auth can-i delete limitranges
no
LimitRange Distribution with TenantReplications
In the future Cluster-Administrators must distribute LimitRanges via TenantReplications. This is a more flexible and powerful way to distribute LimitRanges, as it allows to distribute any kind of resource, not only LimitRanges. Here’s an example of how to distribute a LimitRange to all the Namespaces of a tenant:
apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
name: solar-limitranges
namespace: solar-system
spec:
resyncPeriod: 60s
resources:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: solar
rawItems:
- apiVersion: v1
kind: LimitRange
metadata:
name: cpu-resource-constraint
spec:
limits:
- default: # this section defines default limits
cpu: 500m
defaultRequest: # this section defines default requests
cpu: 500m
max: # max and min define the limit range
cpu: "1"
min:
cpu: 100m
type: Container
PriorityClasses
Pods can have priority. Priority indicates the importance of a Pod relative to other Pods. If a Pod cannot be scheduled, the scheduler tries to preempt (evict) lower priority Pods to make scheduling of the pending Pod possible. See Kubernetes documentation.
In a multi-tenant cluster, not all users can be trusted, as a tenant owner could create Pods at the highest possible priorities, causing other Pods to be evicted/not get scheduled.
To prevent misuses of Pod PriorityClass, Bill, the cluster admin, can enforce the allowed Pod PriorityClass at tenant level:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
priorityClasses:
matchLabels:
env: "production"
With the said Tenant specification, Alice can create a Pod resource if spec.priorityClassName equals to:
- Any
PriorityClass which has the label env with the value production
If a Pod is going to use a non-allowed PriorityClass, it will be rejected by the Validation Webhook enforcing it.
Assign Pod PriorityClass as tenant default
Note: This feature supports type PriorityClass only on API version scheduling.k8s.io/v1
This feature allows specifying a custom default value on a Tenant basis, bypassing the global cluster default (globalDefault=true) that acts only at the cluster level.
It’s possible to assign each Tenant a PriorityClass which will be used, if no PriorityClass is set on pod basis:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
priorityClasses:
default: "tenant-default"
matchLabels:
env: "production"
Let’s create a PriorityClass which is used as the default:
kubectl apply -f - << EOF
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: tenant-default
labels:
env: "production"
value: 1313
preemptionPolicy: Never
globalDefault: false
description: "This is the default PriorityClass for the solar-tenant"
EOF
Note the globalDefault: false which is important to avoid the PriorityClass to be used as the default for all the Tenants. If a Pod has no value for spec.priorityClassName, the default value for PriorityClass (tenant-default) will be used.
RuntimeClasses
Pods can be assigned different RuntimeClasses. With the assigned runtime you can control Container Runtime Interface (CRI) is used for each pod. See Kubernetes documentation for more information.
To prevent misuses of Pod RuntimeClasses, Bill, the cluster admin, can enforce the allowed PodRuntimeClasses at Tenant level:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
runtimeClasses:
matchLabels:
env: "production"
With the said Tenant specification, Alice can create a Pod resource if spec.runtimeClassName equals to:
- Any
RuntimeClass which has the label env with the value production
If a Pod is going to use a non-allowed RuntimeClass, it will be rejected by the Validation Webhook enforcing it.
Assign Runtime Class as tenant default
This feature allows specifying a custom default value on a Tenant basis- It’s possible to assign each tenant a Runtime which will be used, if no Runtime is set on pod basis:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
runtimeClasses:
default: "tenant-default"
matchLabels:
env: "production"
Let’s create a RuntimeClass which is used as the default:
kubectl apply -f - << EOF
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
name: tenant-default
labels:
env: "production"
handler: myconfiguration
EOF
If a Pod has no value for spec.runtimeclass, the default value for RuntimeClass (tenant-default) will be used.
NodeSelector
Bill, the cluster admin, can dedicate a pool of worker nodes to the solar Tenant, to isolate the tenant applications from other noisy neighbors.
These nodes are labeled by Bill as pool=renewable
kubectl get nodes --show-labels
NAME STATUS ROLES AGE VERSION LABELS
...
worker06.acme.com Ready worker 8d v1.25.2 pool=renewable
worker07.acme.com Ready worker 8d v1.25.2 pool=renewable
worker08.acme.com Ready worker 8d v1.25.2 pool=renewable
PodNodeSelector
This approach requires PodNodeSelector Admission Controller plugin to be active. If the plugin is not active, the pods will be scheduled to any node. If your distribution does not support this feature, you can use Expression Node Selectors.
The label pool=renewable is defined as .spec.nodeSelector in the Tenant manifest:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
nodeSelector:
pool: renewable
kubernetes.io/os: linux
The Capsule controller makes sure that any Namespace created in the Tenant has the annotation: scheduler.alpha.kubernetes.io/node-selector: pool=renewable. This annotation tells the scheduler of Kubernetes to assign the node selector pool=renewable to all the Pods deployed in the Tenant. The effect is that all the Pods deployed by Alice are placed only on the designated pool of nodes.
Multiple node selector labels can be defined as in the following snippet:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
nodeSelector:
pool: renewable
kubernetes.io/os: linux
kubernetes.io/arch: amd64
hardware: gpu
Any attempt of Alice to change the selector on the Pods will result in an error from the PodNodeSelector Admission Controller plugin.
kubectl auth can-i edit ns -n solar-production
no
Dynamic resource allocation (DRA)
Dynamic Resource Allocation (DRA) is a Kubernetes capability that allows Pods to request and use shared resources, typically external devices such as hardware accelerators.
See Kubernetes documentation for more information.
Bill can assign a set of dedicated DeviceClasses to tell the solar Tenant what devices they can request.
apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
name: gpu.example.com
labels:
env: "production"
spec:
selectors:
- cel:
expression: device.driver == 'gpu.example.com' && device.attributes['gpu.example.com'].type
== 'gpu'
extendedResourceName: example.com/gpu
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
deviceClasses:
matchLabels:
env: "production"
With the said Tenant specification, Alice can create a ResourceClaim or ResourceClaimTemplate resource if spec.devices.requests[].deviceClassName ( ResourceClaim) or spec.spec.devices.requests[].deviceClassName ( ResourceClaimTemplate) equals to:
- Any DeviceClass, which has the label env with the value production
If any of the devices in the ResourceClaim or ResourceClaimTemplate spec is going to use a non-allowed DeviceClass, the entire request will be rejected by the Validation Webhook enforcing it.
Alice now can create a ResourceClaim using only an allowed DeviceClass:
apiVersion: resource.k8s.io/v1
kind: ResourceClaim
metadata:
name: example-resource-claim
namespace: solar-production
spec:
devices:
requests:
- name: gpu-request
exactly:
deviceClassName: 'gpu.example.com'
Connectivity
Services
ExternalIPs
Specifies the external IPs that can be used in Services with type ClusterIP. An empty list means no IPs are allowed, which is recommended in multi-tenant environments (can be misused for traffic hijacking):
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
serviceOptions:
externalIPs:
allowed: []
Deny labels and annotations
By default, capsule allows Tenant owners to add and modify any label or annotation on their Services.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
serviceOptions:
forbiddenAnnotations:
denied:
- loadbalancer.class.acme.net
deniedRegex: .*.acme.net
forbiddenLabels:
denied:
- loadbalancer.class.acme.net
deniedRegex: .*.acme.net
Deny Service Types
Bill, the cluster admin, can prevent the creation of Services with specific Service types.
NodePort
When dealing with a shared multi-tenant scenario, multiple NodePort services can start becoming cumbersome to manage. The reason behind this could be related to the overlapping needs by the Tenant owners, since a NodePort is going to be open on all nodes and, when using hostNetwork=true, accessible to any Pod although any specific NetworkPolicy.
Bill, the cluster admin, can block the creation of Services with NodePort service type for a given Tenant
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
serviceOptions:
allowedServices:
nodePort: false
With the above configuration, any attempt of Alice to create a Service of type NodePort is denied by the Validation Webhook enforcing it. Default value is true.
ExternalName
Service with the type of ExternalName has been found subject to many security issues. To prevent TenantOwners to create services with the type of ExternalName, the cluster admin can prevent a tenant to create them:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
serviceOptions:
allowedServices:
externalName: false
With the above configuration, any attempt of Alice to create a Service of type externalName is denied by the Validation Webhook enforcing it. Default value is true.
LoadBalancer
Same as previously, the Service of type of LoadBalancer could be blocked for various reasons. To prevent TenantOwners to create these kinds of Services, the cluster admin can Tenant a tenant to create them:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
serviceOptions:
allowedServices:
loadBalancer: false
With the above configuration, any attempt of Alice to create a Service of type LoadBalancer is denied by the Validation Webhook enforcing it. Default value is true.
GatewayClasses
Note: This feature is offered only by API type GatewayClass in group gateway.networking.k8s.io version v1.
GatewayClass is cluster-scoped resource defined by the infrastructure provider. This resource represents a class of Gateways that can be instantiated. Read More
Bill can assign a set of dedicated GatewayClasses to the solar Tenant to force the applications in the solar Tenant to be published only by the assigned Gateway Controller:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
gatewayOptions:
allowedClasses:
matchLabels:
env: "production"
With the said Tenant specification, Alice can create a Gateway resource if spec.gatewayClassName equals to:
- Any
GatewayClass which has the label env with the value production
If an Gateway is going to use a non-allowed GatewayClass, it will be rejected by the Validation Webhook enforcing it.
Alice can create an Gateway using only an allowed GatewayClass:
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
name: example-gateway
namespace: solar-production
spec:
gatewayClassName: customer-class
listeners:
- name: http
protocol: HTTP
port: 80
Any attempt of Alice to use a non-valid GatewayClass, or missing it, is denied by the Validation Webhook enforcing it.
Assign GatewayClass as tenant default
Note: The Default GatewayClass must have a label which is allowed within the tenant. This behavior is only implemented this way for the GatewayClass default.
This feature allows specifying a custom default value on a Tenant basis. Currently there is no global default feature for a GatewayClass. Each Gateway must have a spec.gatewayClassName set.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
gatewayOptions:
allowedClasses:
default: "tenant-default"
matchLabels:
env: "production"
Here’s how the Tenant default GatewayClass could look like:
kubectl apply -f - << EOF
apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
name: tenant-default
labels:
env: "production"
spec:
controllerName: example.com/gateway-controller
EOF
If a Gateway has no value for spec.gatewayClassName, the tenant-default GatewayClass is automatically applied to the Gateway resource.
Ingresses
Assign Ingress Hostnames
Bill can control ingress hostnames in the solar Tenant to force the applications to be published only using the given hostname or set of hostnames:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
ingressOptions:
allowedHostnames:
allowed:
- solar.acmecorp.com
allowedRegex: ^.*acmecorp.com$
The Capsule controller assures that all Ingresses created in the Tenant can use only one of the valid hostnames. Alice can create an Ingress using any allowed hostname:
kubectl apply -f - << EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
namespace: solar-production
spec:
ingressClassName: solar
rules:
- host: web.solar.acmecorp.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx
port:
number: 80
EOF
Any attempt of Alice to use a non-valid hostname is denied by the Validation Webhook enforcing it.
Control Hostname collision in Ingresses
In a multi-tenant environment, as more and more ingresses are defined, there is a chance of collision on the hostname leading to unpredictable behavior of the Ingress Controller. Bill, the cluster admin, can enforce hostname collision detection at different scope levels:
- Cluster
- Tenant
- Namespace
- Disabled (default)
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
- name: joe
kind: User
ingressOptions:
hostnameCollisionScope: Tenant
When a TenantOwner creates an Ingress resource, Capsule will check the collision of hostname in the current ingress with all the hostnames already used, depending on the defined scope.
For example, Alice, one of the TenantOwners, creates an Ingress:
kubectl apply -f - << EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
namespace: solar-production
spec:
rules:
- host: web.solar.acmecorp.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx
port:
number: 80
EOF
Another user, Joe creates an Ingress having the same hostname:
kubectl apply -f - << EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
namespace: solar-development
spec:
rules:
- host: web.solar.acmecorp.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: nginx
port:
number: 80
EOF
When a collision is detected at scope defined by spec.ingressOptions.hostnameCollisionScope, the creation of the Ingress resource will be rejected by the Validation Webhook enforcing it. When spec.ingressOptions.hostnameCollisionScope=Disabled (default), no collision detection is made at all.
Deny Wildcard Hostname in Ingresses
Bill, the cluster admin, can deny the use of wildcard hostname in Ingresses. Let’s assume that Acme Corp. uses the domain acme.com.
As a TenantOwner of solar, Alice creates an Ingress with the host like - host: "*.acme.com". That can lead problems for the water tenant because Alice can deliberately create ingress with host: water.acme.com.
To avoid this kind of problems, Bill can deny the use of wildcard hostnames in the following way:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
ingressOptions:
allowWildcardHostnames: false
Doing this, Alice will not be able to use *.water.acme.com, being the tenant owner of solar and green only.
IngressClasses
An Ingress Controller is used in Kubernetes to publish services and applications outside of the cluster. An Ingress Controller can be provisioned to accept only Ingresses with a given IngressClass.
Bill can assign a set of dedicated IngressClass to the solar Tenant to force the applications in the solar tenant to be published only by the assigned Ingress Controller:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
ingressOptions:
allowedClasses:
matchLabels:
env: "production"
With the said Tenant specification, Alice can create a Ingress resource if spec.ingressClassName or metadata.annotations."kubernetes.io/ingress.class" equals to:
- Any
IngressClass which has the label env with the value production
If an Ingress is going to use a non-allowed IngressClass, it will be rejected by the Validation Webhook enforcing it.
Alice can create an Ingress using only an allowed IngressClass:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: nginx
namespace: solar-production
spec:
ingressClassName: legacy
rules:
- host: solar.acmecorp.com
http:
paths:
- backend:
service:
name: nginx
port:
number: 80
path: /
pathType: ImplementationSpecific
Any attempt of Alice to use a non-valid Ingress Class, or missing it, is denied by the Validation Webhook enforcing it.
Assign Ingress Class as tenant default
Note: This feature is offered only by API type IngressClass in group networking.k8s.io version v1. However, resource Ingress is supported in networking.k8s.io/v1 and networking.k8s.io/v1beta1
This feature allows specifying a custom default value on a Tenant basis, bypassing the global cluster default (with the annotation metadata.annotations.ingressclass.kubernetes.io/is-default-class=true) that acts only at the cluster level. More information: Default IngressClass
It’s possible to assign each Tenant an IngressClass which will be used, if a class is not set on Ingress basis:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
ingressOptions:
allowedClasses:
default: "tenant-default"
matchLabels:
env: "production"
Here’s how the Tenant default IngressClass could look like:
kubectl apply -f - << EOF
apiVersion: networking.k8s.io/v1
kind: IngressClass
metadata:
labels:
env: "production"
app.kubernetes.io/component: controller
name: tenant-default
annotations:
ingressclass.kubernetes.io/is-default-class: "false"
spec:
controller: k8s.io/customer-nginx
EOF
If an Ingress has no value for spec.ingressClassName or metadata.annotations."kubernetes.io/ingress.class", the tenant-default IngressClass is automatically applied to the Ingress resource.
NetworkPolicies
Deprecated
This feature will be deprecated in a future release of Capsule. Instead use TenantReplications. This is also true if you would like other NetworkPolicy implementation like Cilium.
Kubernetes network policies control network traffic between Namespaces and between pods in the same Namespace. Bill, the cluster admin, can enforce network traffic isolation between different Tenants while leaving to Alice, the TenantOwner, the freedom to set isolation between Namespaces in the same Tenant or even between pods in the same Namespace.
To meet this requirement, Bill needs to define network policies that deny pods belonging to Alice’s Namespaces to access pods in Namespaces belonging to other Tenants, e.g. Bob’s Tenant water, or in system Namespaces, e.g. kube-system.
Keep in mind, that because of how the NetworkPolicies API works, the users can still add a policy which contradicts what the Tenant has set, resulting in users being able to circumvent the initial limitation set by the Tenant admin. Two options can be put in place to mitigate this potential privilege escalation: 1. providing a restricted role rather than the default admin one 2. using Calico’s GlobalNetworkPolicy, or Cilium’s CiliumClusterwideNetworkPolicy which are defined at the cluster-level, thus creating an order of packet filtering.
Also, Bill can make sure pods belonging to a Tenant Namespace cannot access other network infrastructures like cluster nodes, load balancers, and virtual machines running other services.
Bill can set network policies in the Tenant manifest, according to the requirements:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
networkPolicies:
items:
- policyTypes:
- Ingress
- Egress
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
except:
- 192.168.0.0/16
ingress:
- from:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: water
- podSelector: {}
- ipBlock:
cidr: 192.168.0.0/16
podSelector: {}
The Capsule controller, watching for Namespace creation, creates the Network Policies for each Namespace in the Tenant.
Alice has access to network policies:
kubectl -n solar-production get networkpolicies
NAME POD-SELECTOR AGE
capsule-solar-0 <none> 42h
Alice can create, patch, and delete additional network policies within her Namespaces:
kubectl -n solar-production auth can-i get networkpolicies
yes
kubectl -n solar-production auth can-i delete networkpolicies
yes
kubectl -n solar-production auth can-i patch networkpolicies
yes
For example, she can create:
kubectl apply -f - << EOF
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
labels:
name: production-network-policy
namespace: solar-production
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress
EOF
Check all the network policies
kubectl -n solar-production get networkpolicies
NAME POD-SELECTOR AGE
capsule-solar-0 <none> 42h
production-network-policy <none> 3m
And delete the Namespace network policies:
kubectl -n solar-production delete networkpolicy production-network-policy
Any attempt of Alice to delete the Tenant network policy defined in the tenant manifest is denied by the Validation Webhook enforcing it. Any deletion by a cluster-administrator will cause the network policy to be recreated by the Capsule controller.
NetworkPolicy Distribution with TenantReplications
In the future Cluster-Administrators must distribute NetworkPolicies via TenantReplications. This is a more flexible and powerful way to distribute NetworkPolicies, as it allows to distribute any kind of resource. Here’s an example of how to distribute a CiliumNetworkPolicy to all the Namespaces of a Tenant:
apiVersion: capsule.clastix.io/v1beta2
kind: TenantResource
metadata:
name: solar-limitranges
namespace: solar-system
spec:
resyncPeriod: 60s
resources:
- namespaceSelector:
matchLabels:
capsule.clastix.io/tenant: solar
rawItems:
- apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
name: "l3-rule"
spec:
endpointSelector:
matchLabels:
role: backend
ingress:
- fromEndpoints:
- matchLabels:
role: frontend
Storage
PersistentVolumes
Any Tenant owner is able to create a PersistentVolumeClaim that, backed by a given StorageClass, will provide volumes for their applications.
In most cases, once a PersistentVolumeClaim is deleted, the bounded PersistentVolume will be recycled due.
However, in some scenarios, the StorageClass or the provisioned PersistentVolume itself could change the retention policy of the volume, keeping it available for recycling and being consumable for another Pod.
In such a scenario, Capsule enforces the Volume mount only to the Namespaces belonging to the Tenant on which it’s been consumed, by adding a label to the Volume as follows.
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
pv.kubernetes.io/provisioned-by: rancher.io/local-path
creationTimestamp: "2022-12-22T09:54:46Z"
finalizers:
- kubernetes.io/pv-protection
labels:
capsule.clastix.io/tenant: solar
name: pvc-1b3aa814-3b0c-4912-9bd9-112820da38fe
resourceVersion: "2743059"
uid: 9836ae3e-4adb-41d2-a416-0c45c2da41ff
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 10Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: melange
namespace: caladan
resourceVersion: "2743014"
uid: 1b3aa814-3b0c-4912-9bd9-112820da38fe
Once the PeristentVolume become available again, it can be referenced by any PersistentVolumeClaim in the solar Tenant Namespace resources.
If another Tenant, like green, tries to use it, it will get an error:
$ kubectl describe pv pvc-9788f5e4-1114-419b-a830-74e7f9a33f5d
Name: pvc-9788f5e4-1114-419b-a830-74e7f9a33f5d
Labels: capsule.clastix.io/tenant=solar
Annotations: pv.kubernetes.io/provisioned-by: rancher.io/local-path
Finalizers: [kubernetes.io/pv-protection]
StorageClass: standard
Status: Available
...
$ cat /tmp/pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: melange
namespace: green-energy
spec:
storageClassName: standard
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 3Gi
volumeName: pvc-9788f5e4-1114-419b-a830-74e7f9a33f5d
$ kubectl apply -f /tmp/pvc.yaml
Error from server: error when creating "/tmp/pvc.yaml": admission webhook "pvc.capsule.clastix.io" denied the request: PeristentVolume pvc-9788f5e4-1114-419b-a830-74e7f9a33f5d cannot be used by the following Tenant, preventing a cross-tenant mount
StorageClasses
Persistent storage infrastructure is provided to Tenants. Different types of storage requirements, with different levels of QoS, eg. SSD versus HDD, are available for different tenants according to the Tenant’s profile. To meet these different requirements, Bill, the cluster admin can provision different StorageClasses and assign them to the tenant:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
storageClasses:
matchLabels:
env: "production"
With the said Tenant specification, Alice can create a Persistent Volume Claims if spec.storageClassName equals to:
- Any
StorageClass which has the label env with the value production
Capsule assures that all PersistentVolumeClaims created by Alice will use only one of the valid storage classes. Assume the StorageClass ceph-rbd has the label env: production:
kubectl apply -f - << EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc
namespace: solar-production
spec:
storageClassName: ceph-rbd
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 12Gi
EOF
If a PersistentVolumeClaim is going to use a non-allowed Storage Class, it will be rejected by the Validation Webhook enforcing it.
Assign Storage Class as tenant default
Note: This feature supports type StorageClass only on API version storage.k8s.io/v1
This feature allows specifying a custom default value on a Tenant basis, bypassing the global cluster default (.metadata.annotations.storageclass.kubernetes.io/is-default-class=true) that acts only at the cluster level. See the Default Storage Class section on Kubernetes documentation.
It’s possible to assign each tenant a StorageClass which will be used, if no value is set on PersistentVolumeClaim basis:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
storageClasses:
default: "tenant-default"
matchLabels:
env: "production"
Here’s how the new StorageClass could look like:
kubectl apply -f - << EOF
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: tenant-default
labels:
env: production
annotations:
storageclass.kubernetes.io/is-default-class: "false"
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
EOF
If a PersistentVolumeClaim has no value for spec.storageClassName the tenant-default value will be used on new PersistentVolumeClaim resources.
Images
PullPolicy
Bill is a cluster admin providing a Container as a Service platform using shared nodes.
Alice, a TenantOwner, can start container images using private images: according to the Kubernetes architecture, the kubelet will download the layers on its cache.
Bob, an attacker, could try to schedule a Pod on the same node where Alice is running her Pods backed by private images: they could start new Pods using ImagePullPolicy=IfNotPresent and be able to start them, even without required authentication since the image is cached on the node.
To avoid this kind of attack, Bill, the cluster admin, can force Alice, the TenantOwner, to start her Pods using only the allowed values for ImagePullPolicy, enforcing the Kubelet to check the authorization first.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
imagePullPolicies:
- Always
Allowed values are: Always, IfNotPresent, Never. As defined by the Kubernetes API
Any attempt of Alice to use a disallowed imagePullPolicies value is denied by the Validation Webhook enforcing it.
Images Registries
Bill, the cluster admin, can set a strict policy on the applications running into Alice’s Tenant: he’d like to allow running just images hosted on a list of specific container registries.
The spec.containerRegistries addresses this task and can provide a combination with hard enforcement using a list of allowed values.
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
containerRegistries:
allowed:
- docker.io
- quay.io
allowedRegex: 'internal.registry.\\w.tld'
In case of Pod running non-FQCI (non fully qualified container image) containers, the container registry enforcement will disallow the execution. If you would like to run a bbusybox:latest container that is commonly hosted on Docker Hub, the TenantOwner has to specify its name explicitly, like docker.io/library/busybox:latest.
A Pod running internal.registry.foo.tld/capsule:latest as registry will be allowed, as well internal.registry.bar.tld since these are matching the regular expression.
A catch-all regex entry as .* allows every kind of registry, which would be the same result of unsetting .spec.containerRegistries at all.
Any attempt of Alice to use a not allowed .spec.containerRegistries value is denied by the Validation Webhook enforcing it.
7 - Metadata
Inherit additional metadata on Tenant resources.
Managed
By default all namespaced resources within a Namespace which are part of a Tenant labeled at admission with the following labels:
capsule.clastix.io/managed-by: <tenant-name> (Legacy label)projectcapsule.dev/tenant: <tenant-name>
The labels are used by Capsule to identify resources belonging to a specific tenant. This is currently important for the Capsule Proxy to filter resources accordingly.
Namespaces
The cluster admin can enforce tenant owners to add specific metadata as Labels and Annotations to the Namespaces they create. This is a useful feature to enforce a set of Rules based on Labels.
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
namespaceOptions:
requiredMetadata:
labels:
env: "^(prod|test|dev)$"
annotations:
example.corp/cost-center: "^INV-[0-9]{4}$"
If you add these properties to a Tenant, and there’s already a Namespace in that Tenant that does not comply with the required metadata, the Namespace will have admission errors until the required metadata is added to it.
Example with Rules:
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
namespaceOptions:
requiredMetadata:
labels:
env: "^(prod|test|dev)$"
annotations:
example.corp/cost-center: "^INV-[0-9]{4}$"
rules:
# Select a subset of namespaces (enviornment=prod) to allow further registries
- namespaceSelector:
matchExpressions:
- key: env
operator: In
values: ["prod"]
enforce:
registries:
- url: "harbor/v2/prod-registry/.*"
policy: [ "ifNotPresent" ]
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
namespaceOptions:
additionalMetadataList:
- annotations:
templated-annotation: {{ tenant.name }}
labels:
templated-label: {{ namespace }}
The cluster admin can “taint” the namespaces created by tenant owners with additional metadata as labels and annotations. There is no specific semantic assigned to these labels and annotations: they will be assigned to the namespaces in the tenant as they are created. However you have the option to be more specific by selecting to which namespaces you want to assign what kind of metadata:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
namespaceOptions:
additionalMetadataList:
# An item without any further selectors is applied to all namspaces
- annotations:
storagelocationtype: s3
labels:
projectcapsule.dev/backup: "true"
# Select a subset of namespaces to apply metadata on
- namespaceSelector:
matchExpressions:
- key: projectcapsule.dev/low_security_profile
operator: NotIn
values: ["true"]
labels:
pod-security.kubernetes.io/enforce: baseline
- namespaceSelector:
matchExpressions:
- key: projectcapsule.dev/low_security_profile
operator: In
values: ["true"]
labels:
pod-security.kubernetes.io/enforce: privileged
Deprecated
This feature is deprecated and will be removed in a future release of Capsule. Migrate to using
AdditionalMetadataListThe cluster admin can “taint” the namespaces created by tenant owners with additional metadata as labels and annotations. There is no specific semantic assigned to these labels and annotations: they will be assigned to the namespaces in the tenant as they are created. This can help the cluster admin to implement specific use cases as, for example, leave only a given tenant to be backed up by a backup service.
Assigns additional labels and annotations to all namespaces created in the solar tenant:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
namespaceOptions:
additionalMetadata:
annotations:
storagelocationtype: s3
labels:
projectcapsule.dev/backup: "true"
When the tenant owner creates a namespace, it inherits the given label and/or annotation:
apiVersion: v1
kind: Namespace
metadata:
annotations:
storagelocationtype: s3
labels:
capsule.clastix.io/tenant: solar
kubernetes.io/metadata.name: solar-production
name: solar-production
projectcapsule.dev/backup: "true"
name: solar-production
ownerReferences:
- apiVersion: capsule.clastix.io/v1beta2
blockOwnerDeletion: true
controller: true
kind: Tenant
name: solar
spec:
finalizers:
- kubernetes
status:
phase: Active
Deny labels and annotations on Namespaces
By default, capsule allows tenant owners to add and modify any label or annotation on their namespaces.
But there are some scenarios, when tenant owners should not have an ability to add or modify specific labels or annotations (for example, this can be labels used in Kubernetes network policies which are added by cluster administrator).
Bill, the cluster admin, can deny Alice to add specific labels and annotations on namespaces:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
namespaceOptions:
forbiddenAnnotations:
denied:
- foo.acme.net
- bar.acme.net
deniedRegex: .*.acme.net
forbiddenLabels:
denied:
- foo.acme.net
- bar.acme.net
deniedRegex: .*.acme.net
owners:
- name: alice
kind: User
Nodes
Warning
Due to
CVE-2021-25735 this feature is only supported for Kubernetes version older than: v1.18.18, v1.19.10, v1.20.6, v1.21.0
When using capsule together with capsule-proxy, Bill can allow Tenant Owners to modify Nodes.
By default, it will allow tenant owners to add and modify any label or annotation on their nodes.
But there are some scenarios, when tenant owners should not have an ability to add or modify specific labels or annotations (there are some types of labels or annotations, which must be protected from modifications - for example, which are set by cloud-providers or autoscalers).
Bill, the cluster admin, can deny Tenant Owners to add or modify specific labels and annotations on Nodes:
apiVersion: capsule.clastix.io/v1beta2
kind: CapsuleConfiguration
metadata:
name: default
spec:
nodeMetadata:
forbiddenAnnotations:
denied:
- foo.acme.net
- bar.acme.net
deniedRegex: .*.acme.net
forbiddenLabels:
denied:
- foo.acme.net
- bar.acme.net
deniedRegex: .*.acme.net
userGroups:
- projectcapsule.dev
- system:serviceaccounts:default
Services
The cluster admin can “taint” the services created by the tenant owners with additional metadata as labels and annotations.
Assigns additional labels and annotations to all services created in the solar tenant:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
serviceOptions:
additionalMetadata:
annotations:
storagelocationtype: s3
labels:
projectcapsule.dev/backup: "true"
When the tenant owner creates a service in a tenant namespace, it inherits the given label and/or annotation:
apiVersion: v1
kind: Service
metadata:
name: nginx
namespace: solar-production
labels:
projectcapsule.dev/backup: "true"
annotations:
storagelocationtype: s3
spec:
ports:
- protocol: TCP
port: 80
targetPort: 8080
selector:
run: nginx
type: ClusterIP
Pods
The cluster admin can “taint” the pods created by the tenant owners with additional metadata as labels and annotations.
Assigns additional labels and annotations to all services created in the solar tenant:
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
owners:
- name: alice
kind: User
podOptions:
additionalMetadata:
annotations:
storagelocationtype: s3
labels:
projectcapsule.dev/backup: "true"
When the tenant owner creates a service in a tenant namespace, it inherits the given label and/or annotation:
apiVersion: v1
kind: Pod
metadata:
name: nginx
namespace: solar-production
labels:
projectcapsule.dev/backup: "true"
annotations:
storagelocationtype: s3
...
8 - Rules
Configure policies and restrictions on a per-Namespace basis with Rules
Enforcement rules allow Bill, the cluster administrator, to set policies and restrictions on a per-Tenant basis. These rules are enforced by Capsule admission webhooks when Alice, the TenantOwner, creates or modifies resources in her Namespaces. With the rule construct, namespaces within the same tenant can be profiled differently depending on their metadata.
8.1 - Enforcement
Configure policies and restrictions and enforce rules per namespace
Namespace rules can enforce admission behavior for selected resources in Tenant namespaces. Each enforce block can define an action and one or more matchers.
Rules are evaluated in declaration order. If multiple allow or deny rules match the same request, the last matching allow or deny rule wins. If at least one allow rule is configured for a workload matcher and no allow or deny rule matches the evaluated value, Capsule denies the request. In other words, allow rules create an allow-list for that matcher. audit rules are purely observational: they never influence the allow/deny decision, but all matching audit rules emit Kubernetes events and add admission warnings.
Action
Each enforce block supports an action field:
| Action | Behavior |
|---|
allow | Allows the matching request and enables allow-list behavior for the matcher. If at least one allow rule exists and no allow or deny rule matches a value, Capsule denies that value. Additional constraints, such as image pull policy, must also be satisfied. |
deny | Denies the matching request. A later matching allow rule can override it. |
audit | Emits a Kubernetes event and returns an admission warning when it matches. It does not allow or deny the request. |
If action is omitted, Capsule treats the rule as deny.
Allow-list behavior is evaluated per workload matcher and per evaluated value. For example, if a registry allow rule exists for harbor/.*, a Pod image from docker.io/library/nginx:latest is denied unless another later or earlier allow rule also matches that image. Audit rules do not satisfy this allow-list requirement.
This precedence model allows both broad defaults and specific exceptions. For example, you can allow all Harbor images but deny a customer path afterwards:
rules:
- enforce:
action: allow
workloads:
registries:
- exp: "harbor/.*"
- enforce:
action: deny
workloads:
registries:
- exp: "harbor/customer/.*"
In this example, harbor/nginx:1.14.2 is allowed, while harbor/customer/app:1.0.0 is denied because the later, more specific deny rule also matches.
You can also deny broadly and allow a more specific exception afterwards:
rules:
- enforce:
action: deny
workloads:
registries:
- exp: "harbor/customer/.*"
- enforce:
action: allow
workloads:
registries:
- exp: "harbor/customer/prod-image/.*"
In this example, harbor/customer/test-image/app:1.0.0 is denied, while harbor/customer/prod-image/app:1.0.0 is allowed.
Match expressions
Several workload rule types use a common match expression structure. A matcher must define at least one of exact or exp. Both fields may be set together; in that case, the matcher succeeds when either the exact list or the regular expression matches.
exact:
- value-a
- value-b
exp: "value-[0-9]+"
| Field | Description |
|---|
exact | A list of exact values. The matcher succeeds when the evaluated value equals one of the listed values. |
exp | A regular expression matched against the evaluated value. |
negate | Negates the final match result. This applies to both exact and exp. |
For example, this matcher matches registry.local/team-a/app:1.0.0, registry.local/team-b/app:1.0.0, or any reference under registry.local/shared/*:
exact:
- registry.local/team-a/app:1.0.0
- registry.local/team-b/app:1.0.0
exp: "registry.local/shared/.*"
With negate: true, the final match result is inverted. This means negation applies to exact values as well as regular expressions:
exact:
- registry.local/blocked/app:1.0.0
exp: "registry.local/deprecated/.*"
negate: true
This matcher succeeds for every value except registry.local/blocked/app:1.0.0 and values matching registry.local/deprecated/.*.
Audit
Use action: audit to observe workload usage without directly blocking the request. Audit rules emit Kubernetes events and add warnings to the admission response, but they do not allow or deny the request. If an allow-list is active for the same matcher and no allow rule matches the evaluated value, the request is still denied even when an audit rule matches.
For registry enforcement:
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
...
rules:
- enforce:
action: audit
workloads:
targets:
- pod/containers
registries:
- exp: "docker.io/.*"
Applying a Pod with docker.io/library/nginx:latest succeeds in this audit-only example because no registry allow-list is configured. The API server response contains an admission warning and Capsule emits a related event for the Pod.
For QoS enforcement:
rules:
- enforce:
action: audit
workloads:
qosClasses:
- Burstable
Applying a Burstable Pod succeeds in this audit-only example because no QoS allow-list is configured. Capsule emits an event and returns an admission warning.
For scheduler enforcement:
rules:
- enforce:
action: audit
workloads:
schedulers:
- exact:
- custom-scheduler
Applying a Pod with spec.schedulerName: custom-scheduler succeeds in this audit-only example because no scheduler allow-list is configured. Capsule emits an audit event and returns an admission warning.
When audit rules are used together with allow rules, the matching value must still be allowed explicitly. For example, an audited registry reference that does not match any registry allow rule is denied by the allow-list, but Capsule still emits the audit event before denying the request.
Workloads
Enforcement for workloads mainly targets Pods and their associated resources.
Workload enforcement is configured under spec.rules[].enforce.workloads. Each rule can define an action, optional workload targets, and one or more workload matchers such as registry match expressions, scheduler match expressions, or QoS classes.
QoS Classes
QoS class enforcement allows administrators to allow, deny, or audit Pods based on their computed Kubernetes QoS class.
QoS rules are configured under enforce.workloads.qosClasses.
Supported QoS classes are:
| QoS class | Description |
|---|
Guaranteed | The Pod has CPU and memory requests and limits set so that requests equal limits. |
Burstable | The Pod has at least one CPU or memory request or limit, but does not qualify as Guaranteed. |
BestEffort | The Pod has no CPU or memory requests or limits. |
Capsule evaluates the QoS class of the incoming Pod during create and update admission. If Kubernetes has already populated status.qosClass, Capsule can use that value; otherwise it computes the QoS class from the Pod specification.
Deny BestEffort Pods:
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
...
rules:
- enforce:
action: deny
workloads:
qosClasses:
- BestEffort
With this rule, a Pod without CPU or memory requests and limits is denied:
apiVersion: v1
kind: Pod
metadata:
name: best-effort
spec:
containers:
- name: shell
image: harbor/platform/debian:latest
command: ["sleep", "infinity"]
Example rejection:
Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: QoS class "BestEffort" at status.qosClass is denied by namespace rule
Audit Burstable Pods:
rules:
- enforce:
action: audit
workloads:
qosClasses:
- Burstable
A matching Pod is admitted in this audit-only example, but Capsule emits an event and the API server response contains an admission warning. If a QoS allow-list is also configured and the Pod’s QoS class is not allowed, the Pod is denied while the audit event is still emitted.
Allow BestEffort only for selected namespaces:
rules:
- enforce:
action: deny
workloads:
qosClasses:
- BestEffort
- namespaceSelector:
matchLabels:
allow-best-effort: "true"
enforce:
action: allow
workloads:
qosClasses:
- BestEffort
Because later matching allow or deny rules take precedence, namespaces labeled allow-best-effort=true can run BestEffort Pods, while other namespaces cannot.
Scheduler Names
Scheduler enforcement allows administrators to allow, deny, or audit Pods based on spec.schedulerName.
Scheduler rules are configured under enforce.workloads.schedulers. Each scheduler matcher uses the common match expression structure with exact, exp, and optional negate.
Capsule evaluates spec.schedulerName during Pod create and update admission. If spec.schedulerName is empty or omitted, scheduler enforcement does not match it and does not normalize it to default-scheduler.
Allow only selected explicit schedulers:
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
...
rules:
- enforce:
action: allow
workloads:
schedulers:
- exact:
- tenant-scheduler
- batch-scheduler
A Pod using one of the listed schedulers is admitted:
apiVersion: v1
kind: Pod
metadata:
name: scheduled-by-tenant
spec:
schedulerName: tenant-scheduler
containers:
- name: shell
image: harbor/platform/debian:latest
command: ["sleep", "infinity"]
A Pod using another explicit scheduler is denied:
apiVersion: v1
kind: Pod
metadata:
name: scheduled-by-other
spec:
schedulerName: other-scheduler
containers:
- name: shell
image: harbor/platform/debian:latest
command: ["sleep", "infinity"]
Example rejection:
Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: scheduler "other-scheduler" at spec.schedulerName is not allowed by namespace rule
Use a regular expression to allow a scheduler family:
rules:
- enforce:
action: allow
workloads:
schedulers:
- exp: "tenant-[a-z0-9-]+"
Use exact and exp together to allow a fixed list plus a pattern:
rules:
- enforce:
action: allow
workloads:
schedulers:
- exact:
- default-scheduler
- batch-scheduler
exp: "tenant-[a-z0-9-]+"
This matcher allows default-scheduler, batch-scheduler, and scheduler names matching tenant-[a-z0-9-]+.
Deny a known unsafe scheduler:
rules:
- enforce:
action: deny
workloads:
schedulers:
- exact:
- unsafe-scheduler
Use negate: true to deny every explicit scheduler except a trusted set:
rules:
- enforce:
action: deny
workloads:
schedulers:
- exact:
- default-scheduler
- tenant-scheduler
negate: true
Because negate applies to exact, this rule matches any explicit scheduler name except default-scheduler and tenant-scheduler.
Audit usage of a custom scheduler:
rules:
- enforce:
action: audit
workloads:
schedulers:
- exact:
- custom-scheduler
A matching Pod is admitted in this audit-only example, but Capsule emits an audit event and returns an admission warning. If a scheduler allow-list is also configured and the scheduler name is not allowed, the Pod is denied while the audit event is still emitted.
OCI Registries
Registry enforcement allows administrators to allow, deny, or audit Pod image references. Registry matchers are evaluated against the full OCI reference string, including registry, repository path, image name, tag, or digest.
Registry rules are configured under enforce.workloads.registries. The workload-level targets field under enforce.workloads.targets controls which Pod image references are validated.
Registry matchers use the common match expression structure:
registries:
- exact:
- harbor/platform/debian:latest
- harbor/platform/busybox:latest
- exp: "harbor/platform/.*"
Use exact for a fixed list of complete references and exp for path or registry patterns. A single matcher may contain both fields:
registries:
- exact:
- harbor/platform/debian:latest
exp: "harbor/shared/.*"
This matcher succeeds for harbor/platform/debian:latest or any reference matching harbor/shared/.*.
The following example allows Harbor images by default, denies a more specific customer path for regular containers and image volumes, allows and audits regular container images from an audit registry, and allows a production image path only for namespaces matching env=prod:
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
...
rules:
- enforce:
action: allow
workloads:
registries:
- exp: "harbor/.*"
- enforce:
action: deny
workloads:
targets:
- pod/containers
- pod/volumes
registries:
- exp: "harbor/customer/.*"
- enforce:
action: allow
workloads:
targets:
- pod/containers
registries:
- exp: "audit/.*"
- enforce:
action: audit
workloads:
targets:
- pod/containers
registries:
- exp: "audit/.*"
- namespaceSelector:
matchExpressions:
- key: env
operator: In
values: ["prod"]
enforce:
action: allow
workloads:
targets:
- pod/containers
- pod/volumes
registries:
- exp: "harbor/customer/prod-image/.*"
policy: ["Always"]
Apply the following Pod in namespace solar-test, which does not match the env=prod selector:
apiVersion: v1
kind: Pod
metadata:
name: image-volume
spec:
containers:
- name: shell
command: ["sleep", "infinity"]
imagePullPolicy: IfNotPresent
image: harbor/customer/test-image/debian:latest
volumeMounts:
- name: volume
mountPath: /volume
volumes:
- name: volume
image:
reference: quay.io/crio/artifact:v2
pullPolicy: IfNotPresent
The request is denied:
kubectl apply -f pod.yaml -n solar-test
Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: containers[0] reference "harbor/customer/test-image/debian:latest" is denied by registry rule "harbor/customer/.*"
The Pod is denied because the regular container image matches both harbor/.* and harbor/customer/.*. Since the deny rule is declared later, it has higher precedence.
The image volume reference is not denied by the shown deny rule because it does not match harbor/customer/.*. If the image volume used a matching reference, for example harbor/customer/volume-artifact:v1, the same deny rule would apply because it targets both pod/containers and pod/volumes.
In a namespace matching env=prod, the more specific production allow rule is also considered:
apiVersion: v1
kind: Pod
metadata:
name: prod-image
spec:
containers:
- name: shell
command: ["sleep", "infinity"]
imagePullPolicy: Always
image: harbor/customer/prod-image/debian:latest
The request is allowed because the namespace-specific rule matches later and allows harbor/customer/prod-image/.* with imagePullPolicy: Always.
Target-specific registry rules allow different behavior for different parts of the same Pod. For example, this rule denies the registry only for init containers:
rules:
- enforce:
action: deny
workloads:
targets:
- pod/initcontainers
registries:
- exp: "harbor/init-only/.*"
A matching reference under spec.initContainers is denied. The same reference under spec.containers is ignored by this rule.
Registry exact match examples
Use exact when you want to allow or deny a fixed set of complete image references:
rules:
- enforce:
action: allow
workloads:
targets:
- pod/containers
registries:
- exact:
- harbor/platform/debian:latest
- harbor/platform/busybox:1.36
A Pod using harbor/platform/debian:latest or harbor/platform/busybox:1.36 is admitted. A Pod using harbor/platform/nginx:latest is denied because an allow rule exists for registry enforcement but does not match that reference.
You can combine exact and exp in the same registry matcher:
rules:
- enforce:
action: allow
workloads:
registries:
- exact:
- harbor/platform/debian:latest
exp: "harbor/shared/.*"
This rule allows the exact Debian image and any image under harbor/shared/*.
PullPolicy
Define the allowed image pull policies for a matching registry rule. Supported policies are:
Always: The image is always pulled.IfNotPresent: The image is pulled only if it is not already present on the node.Never: The image is never pulled. If the image is not present on the node, the Pod fails to start.
The policy field is optional. If no policy is specified, all image pull policies are accepted for the matching registry rule.
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
...
rules:
- enforce:
action: allow
workloads:
targets:
- pod/containers
registries:
- exp: "harbor/v2/customer-registry/.*"
policy: ["IfNotPresent", "Always"]
If the final matching registry decision is allow and that matching registry rule defines policy, the Pod must use one of the configured pull policies. For example, this rule allows the registry but only with Always:
rules:
- enforce:
action: allow
workloads:
targets:
- pod/containers
registries:
- exp: "harbor/v2/customer-registry/.*"
policy: ["Always"]
A Pod using imagePullPolicy: Never for that registry is rejected:
Error from server (Forbidden): error when creating "pod.yaml": admission webhook "pods.projectcapsule.dev" denied the request: containers[0] reference "harbor/v2/customer-registry/debian:latest" uses pullPolicy=Never which is not allowed (allowed: Always)
Policy is checked only after the final registry decision is allow. A final deny decision always denies the request, regardless of the configured pull policy.
Negation
A registry matcher can be negated with negate: true. Negation applies to the final result of the matcher, including both exact and exp.
For example, the following rule denies every regular container image that is not from the trusted registry path:
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
...
rules:
- enforce:
action: deny
workloads:
targets:
- pod/containers
registries:
- exp: "trusted/.*"
negate: true
With this rule:
trusted/backend/api:1.0.0 is allowed in this deny-only example because it does not match the negated deny rule and no registry allow-list is configured.docker.io/library/nginx:latest is denied because it does not match trusted/.*, so the negated matcher evaluates to true.
Negation also applies to exact values:
rules:
- enforce:
action: deny
workloads:
targets:
- pod/containers
registries:
- exact:
- trusted/backend/api:1.0.0
- trusted/frontend/web:1.0.0
negate: true
This rule denies every explicit container image except the two exact references listed, as long as no separate registry allow-list requires an explicit allow. If an allow rule is configured for the same matcher scope, the excepted references must also match an allow rule.
You can combine exact values, regular expressions, negation, namespace selectors, and action precedence. For example, deny all untrusted container images by default, but allow a controlled exception in production namespaces:
rules:
- enforce:
action: deny
workloads:
targets:
- pod/containers
registries:
- exact:
- trusted/base/debian:latest
exp: "trusted/platform/.*"
negate: true
- enforce:
action: allow
workloads:
targets:
- pod/containers
registries:
- exact:
- trusted/base/debian:latest
exp: "trusted/platform/.*"
- namespaceSelector:
matchLabels:
env: prod
enforce:
action: allow
workloads:
targets:
- pod/containers
registries:
- exp: "partner-registry/prod-approved/.*"
The second rule explicitly allows the trusted references that were excluded from the negated deny rule, which is required when registry allow-list behavior is active. In a namespace labeled env=prod, partner-registry/prod-approved/app:1.0.0 is allowed because the later matching allow rule overrides the earlier negated deny rule.
Targets
The targets field defines which parts of a workload a rule applies to.
Targets are configured under enforce.workloads.targets and are authoritative for target-aware workload enforcement. Registry entries do not define their own validation targets.
rules:
- enforce:
action: deny
workloads:
targets:
- pod/containers
registries:
- exp: "harbor/customer/.*"
If targets is omitted or empty, the rule applies to all workload targets supported by the matching hook.
Supported workload targets are:
| Target | Description |
|---|
pod/initcontainers | Applies to images used by spec.initContainers. |
pod/containers | Applies to images used by spec.containers. |
pod/ephemeralcontainers | Applies to images used by spec.ephemeralContainers. |
pod/volumes | Applies to image volumes under spec.volumes[].image. |
Targets are currently used only by a subset of workload hooks. For example, the registry enforcement hook uses targets to decide which Pod image references are validated. Other hooks may ignore targets until they explicitly support target-aware enforcement.
Examples:
rules:
- enforce:
action: deny
workloads:
targets:
- pod/initcontainers
registries:
- exp: "harbor/init-only/.*"
This rule denies matching images only when they are used by initContainers. The same image reference is not denied when used by regular containers, ephemeral containers, or image volumes unless another rule matches those targets.
rules:
- enforce:
action: deny
workloads:
targets:
- pod/containers
- pod/ephemeralcontainers
registries:
- exp: "debug/.*"
This rule applies to regular containers and ephemeral containers, but not to init containers or image volume
Services
Service enforcement allows administrators to allow, deny, or audit Kubernetes Service resources in Tenant namespaces.
Service rules are configured under spec.rules[].enforce.services. Each rule can define an action, a list of allowed or denied Service types, and optional type-specific constraints for LoadBalancer, ExternalName, and NodePort Services.
rules:
- enforce:
action: allow
services:
types:
- ClusterIP
- NodePort
- LoadBalancer
- ExternalName
loadBalancers:
cidrs:
- 10.0.0.2/32
externalNames:
hostnames:
- exp: ".*\\.example\\.com"
exact:
- internal.git.com
nodePorts:
ports:
- from: 30000
to: 32767
Service enforcement follows the same action and precedence model as other namespace rules:
allow creates an allow-list for the evaluated Service value.deny denies matching values.audit emits events and admission warnings but does not allow or deny the request.- If multiple
allow or deny rules match the same value, the last matching allow or deny rule wins. - If at least one
allow rule exists for a Service matcher and no allow or deny rule matches the evaluated value, Capsule denies the request. - Audit rules never satisfy allow-list behavior.
Service rules are evaluated during Service create and update admission.
Service Types
The services.types field controls which Kubernetes Service types are allowed, denied, or audited by a rule.
Supported values are:
| Type | Description |
|---|
ClusterIP | Allows, denies, or audits Services of type ClusterIP. |
NodePort | Allows, denies, or audits Services of type NodePort. |
LoadBalancer | Allows, denies, or audits Services of type LoadBalancer. |
ExternalName | Allows, denies, or audits Services of type ExternalName. |
Allow only ClusterIP Services:
rules:
- enforce:
action: allow
services:
types:
- ClusterIP
With this rule, a ClusterIP Service is admitted:
apiVersion: v1
kind: Service
metadata:
name: internal-api
spec:
type: ClusterIP
ports:
- name: http
port: 8080
targetPort: 8080
A Service of another type, for example ExternalName, is denied because an allow-list exists for Service types and ExternalName is not listed:
apiVersion: v1
kind: Service
metadata:
name: external-api
spec:
type: ExternalName
externalName: internal.git.com
ports:
- name: http
port: 443
targetPort: 443
Example rejection:
Error from server (Forbidden): error when creating "svc.yaml": admission webhook "services.validating.projectcapsule.dev" denied the request: service type "ExternalName" at spec.type is not allowed by namespace rule: value did not match any allowed rule. Allowed service types: ClusterIP
Deny LoadBalancer Services:
rules:
- enforce:
action: deny
services:
types:
- LoadBalancer
Allow ClusterIP and ExternalName, but deny ExternalName again for selected namespaces:
rules:
- enforce:
action: allow
services:
types:
- ClusterIP
- ExternalName
- namespaceSelector:
matchLabels:
external-services: blocked
enforce:
action: deny
services:
types:
- ExternalName
Because later matching allow or deny decisions win, namespaces labeled external-services=blocked cannot create ExternalName Services, while other matching namespaces can.
Important caveats for services.types
The services.types field is the Service capability gate. Type-specific sections such as loadBalancers, externalNames, and nodePorts do not automatically allow a Service type by themselves.
For example, this rule restricts LoadBalancer CIDRs, but it does not by itself allow LoadBalancer Services if another type allow-list exists that excludes LoadBalancer:
rules:
- enforce:
action: allow
services:
types:
- ClusterIP
- enforce:
action: allow
services:
loadBalancers:
cidrs:
- 10.0.0.2/32
In this example, a LoadBalancer Service is denied by the Service type allow-list because LoadBalancer is not included in services.types.
To allow and constrain LoadBalancer Services, configure both:
rules:
- enforce:
action: allow
services:
types:
- LoadBalancer
loadBalancers:
cidrs:
- 10.0.0.2/32
LoadBalancer
LoadBalancer rules allow administrators to restrict the IPs and source ranges used by Services of type LoadBalancer.
LoadBalancer constraints are configured under enforce.services.loadBalancers.cidrs.
Capsule evaluates the following Service fields:
| Field | Description |
|---|
spec.loadBalancerIP | Explicit LoadBalancer IP requested by the Service. |
spec.loadBalancerSourceRanges[] | Source CIDR ranges allowed to access the LoadBalancer. |
Allow LoadBalancer Services only with a specific IP:
rules:
- enforce:
action: allow
services:
types:
- LoadBalancer
loadBalancers:
cidrs:
- 10.0.0.2/32
This Service is admitted:
apiVersion: v1
kind: Service
metadata:
name: public-api
spec:
type: LoadBalancer
loadBalancerIP: 10.0.0.2
ports:
- name: http
port: 80
targetPort: 8080
This Service is denied because the requested IP is outside the allowed CIDR:
apiVersion: v1
kind: Service
metadata:
name: public-api
spec:
type: LoadBalancer
loadBalancerIP: 10.0.171.239
ports:
- name: http
port: 80
targetPort: 8080
Example rejection:
Error from server (Forbidden): error when creating "svc.yaml": admission webhook "services.validating.projectcapsule.dev" denied the request: loadBalancer CIDR "10.0.171.239" at spec.loadBalancerIP is not allowed by namespace rule: value did not match any allowed rule. Allowed CIDRs: 10.0.0.2/32
Allow a LoadBalancer IP range:
rules:
- enforce:
action: allow
services:
types:
- LoadBalancer
loadBalancers:
cidrs:
- 10.0.1.0/24
The following Service is admitted because 10.0.1.44 is contained in 10.0.1.0/24:
apiVersion: v1
kind: Service
metadata:
name: public-api
spec:
type: LoadBalancer
loadBalancerIP: 10.0.1.44
ports:
- name: http
port: 80
targetPort: 8080
Restrict loadBalancerSourceRanges:
rules:
- enforce:
action: allow
services:
types:
- LoadBalancer
loadBalancers:
cidrs:
- 10.0.1.0/24
This Service is admitted because the requested source range is fully contained in the allowed CIDR:
apiVersion: v1
kind: Service
metadata:
name: public-api
spec:
type: LoadBalancer
loadBalancerSourceRanges:
- 10.0.1.0/25
ports:
- name: http
port: 80
targetPort: 8080
This Service is denied because the requested source range is not fully contained in the allowed CIDR:
apiVersion: v1
kind: Service
metadata:
name: public-api
spec:
type: LoadBalancer
loadBalancerSourceRanges:
- 10.0.1.0/23
ports:
- name: http
port: 80
targetPort: 8080
If any matching rule configures loadBalancers.cidrs, then a LoadBalancer Service must explicitly set at least one of:
spec.loadBalancerIPspec.loadBalancerSourceRanges
This is intentional. If CIDR restrictions are configured, Capsule requires the Service request to provide a value that can be evaluated.
For example, this Service is denied when loadBalancers.cidrs is configured:
apiVersion: v1
kind: Service
metadata:
name: public-api
spec:
type: LoadBalancer
ports:
- name: http
port: 80
targetPort: 8080
Example rejection:
Error from server (Forbidden): error when creating "svc.yaml": admission webhook "services.validating.projectcapsule.dev" denied the request: loadBalancer service requires spec.loadBalancerIP or spec.loadBalancerSourceRanges because loadBalancer CIDR constraints are enforced by namespace rule
If no loadBalancers.cidrs constraint is configured, Capsule does not require these fields. In that case, a LoadBalancer Service can be admitted as long as the Service type itself is allowed.
Denying selected LoadBalancer CIDRs
You can also deny specific LoadBalancer CIDRs:
rules:
- enforce:
action: allow
services:
types:
- LoadBalancer
loadBalancers:
cidrs:
- 10.0.0.0/8
- enforce:
action: deny
services:
loadBalancers:
cidrs:
- 10.0.66.0/24
A Service using 10.0.66.10 is denied because the later deny rule matches:
Error from server (Forbidden): error when creating "svc.yaml": admission webhook "services.validating.projectcapsule.dev" denied the request: loadBalancer CIDR "10.0.66.10" at spec.loadBalancerIP is denied by namespace rule: 10.0.66.10 is contained in 10.0.66.0/24
A later namespace-specific allow rule can override an earlier allow miss or deny decision:
rules:
- enforce:
action: allow
services:
types:
- LoadBalancer
loadBalancers:
cidrs:
- 10.0.0.2/32
- namespaceSelector:
matchLabels:
environment: prod
enforce:
action: allow
services:
loadBalancers:
cidrs:
- 10.0.171.0/24
In namespaces labeled environment=prod, a Service using 10.0.171.239 is admitted. In other namespaces, it is denied because it does not match the default allowed CIDR.
ExternalName
ExternalName rules allow administrators to restrict spec.externalName for Services of type ExternalName.
ExternalName constraints are configured under enforce.services.externalNames.hostnames.
Each hostname matcher uses the common match expression structure with exact, exp, and optional negate.
Allow selected ExternalName hostnames:
rules:
- enforce:
action: allow
services:
types:
- ExternalName
externalNames:
hostnames:
- exact:
- internal.git.com
- exp: ".*\\.example\\.com"
The following Services are admitted:
apiVersion: v1
kind: Service
metadata:
name: git
spec:
type: ExternalName
externalName: internal.git.com
ports:
- name: https
port: 443
targetPort: 443
apiVersion: v1
kind: Service
metadata:
name: api
spec:
type: ExternalName
externalName: api.example.com
ports:
- name: https
port: 443
targetPort: 443
A non-matching hostname is denied:
apiVersion: v1
kind: Service
metadata:
name: api
spec:
type: ExternalName
externalName: api.bad.com
ports:
- name: https
port: 443
targetPort: 443
Example rejection:
Error from server (Forbidden): error when creating "svc.yaml": admission webhook "services.validating.projectcapsule.dev" denied the request: externalName hostname "api.bad.com" at spec.externalName is not allowed by namespace rule: value did not match any allowed rule. Allowed hostnames: exact: internal.git.com, exp: .*\.example\.com
Use exact and exp together in the same matcher:
rules:
- enforce:
action: allow
services:
types:
- ExternalName
externalNames:
hostnames:
- exact:
- combined.internal.git.com
exp: "combined\\..*\\.example\\.com"
This matcher allows both:
combined.internal.git.com- hostnames matching
combined\\..*\\.example\\.com
Negation for ExternalName hostnames
negate: true inverts the final matcher result. This applies to both exact and exp.
Deny every ExternalName except trusted hostnames:
rules:
- enforce:
action: deny
services:
externalNames:
hostnames:
- exp: "trusted\\..*"
negate: true
- enforce:
action: allow
services:
types:
- ExternalName
externalNames:
hostnames:
- exp: "trusted\\..*"
With these rules:
trusted.api is admitted.api.example.com is denied by the negated deny rule.
Example rejection:
Error from server (Forbidden): error when creating "svc.yaml": admission webhook "services.validating.projectcapsule.dev" denied the request: externalName hostname "api.example.com" at spec.externalName is denied by namespace rule: "api.example.com" matched hostname rule not exp: trusted\..*
Important: when an allow-list exists for ExternalName hostnames, values excluded from a negated deny rule still need a matching allow rule. The deny rule prevents untrusted values, while the allow rule satisfies allow-list behavior for trusted values.
Namespace-specific ExternalName rules
You can use namespaceSelector to apply ExternalName restrictions only to selected namespaces:
rules:
- enforce:
action: allow
services:
types:
- ExternalName
externalNames:
hostnames:
- exp: ".*\\.example\\.com"
- namespaceSelector:
matchLabels:
external-policy: restricted
enforce:
action: deny
services:
externalNames:
hostnames:
- exact:
- blocked.example.com
In namespaces labeled external-policy=restricted, blocked.example.com is denied. Other hostnames matching .*\\.example\\.com remain allowed.
NodePort
NodePort rules allow administrators to restrict explicitly requested spec.ports[].nodePort values.
NodePort constraints are configured under enforce.services.nodePorts.ports.
Each port range contains:
| Field | Description |
|---|
from | First allowed or denied port in the range. |
to | Last allowed or denied port in the range. |
The from value must be lower than or equal to to. Equal values are valid and represent a single port.
Allow selected NodePort ranges:
rules:
- enforce:
action: allow
services:
types:
- NodePort
nodePorts:
ports:
- from: 30000
to: 30100
- from: 30500
to: 30500
This Service is admitted because 30080 is in the allowed range:
apiVersion: v1
kind: Service
metadata:
name: tenant-api
spec:
type: NodePort
ports:
- name: http
port: 8080
targetPort: 8080
nodePort: 30080
This Service is also admitted because 30500 matches the single-port range:
apiVersion: v1
kind: Service
metadata:
name: tenant-api-single
spec:
type: NodePort
ports:
- name: http
port: 8080
targetPort: 8080
nodePort: 30500
This Service is denied because 32080 is outside the allowed ranges:
apiVersion: v1
kind: Service
metadata:
name: tenant-api
spec:
type: NodePort
ports:
- name: http
port: 8080
targetPort: 8080
nodePort: 32080
Example rejection:
Error from server (Forbidden): error when creating "svc.yaml": admission webhook "services.validating.projectcapsule.dev" denied the request: nodePort "32080" at spec.ports[0].nodePort is not allowed by namespace rule: value did not match any allowed rule. Allowed ranges: 30000-30100, 30500
If any matching rule configures nodePorts.ports, then a NodePort Service must explicitly set spec.ports[].nodePort.
This is intentional. Kubernetes can allocate a node port automatically when the field is omitted, but the validating webhook cannot know the allocated value at admission time. To enforce configured port ranges reliably, Capsule requires the requested node port to be explicit.
The following Service is denied when nodePorts.ports is configured:
apiVersion: v1
kind: Service
metadata:
name: tenant-api
spec:
type: NodePort
ports:
- name: http
port: 8080
targetPort: 8080
Example rejection:
Error from server (Forbidden): error when creating "svc.yaml": admission webhook "services.validating.projectcapsule.dev" denied the request: service requires explicit spec.ports[*].nodePort because nodePort ranges are enforced by namespace rule
If no nodePorts.ports constraint is configured, Capsule does not require explicit nodePort values. In that case, a NodePort Service can be admitted as long as the Service type itself is allowed.
Denying selected NodePorts
You can allow a broad range and deny a specific port afterwards:
rules:
- enforce:
action: allow
services:
types:
- NodePort
nodePorts:
ports:
- from: 30000
to: 30100
- enforce:
action: deny
services:
nodePorts:
ports:
- from: 30090
to: 30090
A Service using 30080 is admitted. A Service using 30090 is denied because the later deny rule also matches.
Example rejection:
Error from server (Forbidden): error when creating "svc.yaml": admission webhook "services.validating.projectcapsule.dev" denied the request: nodePort "30090" at spec.ports[0].nodePort is denied by namespace rule: nodePort 30090 is within allowed range 30090
Although the detail says the port is within the matched range, the rule action is deny, so the request is rejected.
LoadBalancer Services and NodePorts
Kubernetes LoadBalancer Services may allocate node ports unless spec.allocateLoadBalancerNodePorts is explicitly set to false.
Therefore, NodePort range enforcement also applies to LoadBalancer Services when node port allocation is enabled.
This rule allows LoadBalancer Services, restricts the LoadBalancer IP, and restricts the allocated node port:
rules:
- enforce:
action: allow
services:
types:
- LoadBalancer
loadBalancers:
cidrs:
- 10.0.0.2/32
nodePorts:
ports:
- from: 30000
to: 30100
This Service is admitted because the LoadBalancer IP and node port are both allowed:
apiVersion: v1
kind: Service
metadata:
name: public-api
spec:
type: LoadBalancer
loadBalancerIP: 10.0.0.2
ports:
- name: http
port: 80
targetPort: 8080
nodePort: 30080
This Service is denied because the explicit node port is outside the allowed range:
apiVersion: v1
kind: Service
metadata:
name: public-api
spec:
type: LoadBalancer
loadBalancerIP: 10.0.0.2
ports:
- name: http
port: 80
targetPort: 8080
nodePort: 32080
When nodePorts.ports is configured and LoadBalancer node port allocation is enabled, Capsule requires explicit spec.ports[].nodePort values:
apiVersion: v1
kind: Service
metadata:
name: public-api
spec:
type: LoadBalancer
loadBalancerIP: 10.0.0.2
ports:
- name: http
port: 80
targetPort: 8080
Example rejection:
Error from server (Forbidden): error when creating "svc.yaml": admission webhook "services.validating.projectcapsule.dev" denied the request: service requires explicit spec.ports[*].nodePort because nodePort ranges are enforced by namespace rule
To avoid node port enforcement for a LoadBalancer Service, disable node port allocation explicitly:
apiVersion: v1
kind: Service
metadata:
name: public-api
spec:
type: LoadBalancer
allocateLoadBalancerNodePorts: false
loadBalancerIP: 10.0.0.2
ports:
- name: http
port: 80
targetPort: 8080
With allocateLoadBalancerNodePorts: false, Capsule does not require or validate spec.ports[].nodePort for that LoadBalancer Service. The Service must still satisfy any configured LoadBalancer CIDR rules.
Advanced
Auditing Services
Use action: audit to observe Service usage without directly blocking the request. Audit rules emit Kubernetes events and return admission warnings, but they do not allow or deny the request.
Audit ExternalName usage:
rules:
- enforce:
action: audit
services:
types:
- ExternalName
externalNames:
hostnames:
- exp: "audit\\..*"
A matching Service is admitted in this audit-only example because no Service type or hostname allow-list is configured:
apiVersion: v1
kind: Service
metadata:
name: audited-external
spec:
type: ExternalName
externalName: audit.internal
ports:
- name: https
port: 443
targetPort: 443
If an allow-list is also configured, audit does not satisfy it:
rules:
- enforce:
action: audit
services:
externalNames:
hostnames:
- exp: "audit\\..*"
- enforce:
action: allow
services:
types:
- ExternalName
externalNames:
hostnames:
- exp: "allowed\\..*"
With these rules, audit.internal emits an audit event but is still denied because it does not match the allowed hostname rule.
Combining Service Rules
Service rules can be split across multiple rule blocks. This is useful when type permissions, LoadBalancer CIDR rules, hostname rules, and NodePort ranges should be managed independently.
For example:
rules:
- enforce:
action: allow
services:
types:
- ClusterIP
- ExternalName
- enforce:
action: allow
services:
externalNames:
hostnames:
- exp: ".*\\.example\\.com"
This configuration:
- allows
ClusterIP Services; - allows
ExternalName Services as a type; - allows only ExternalName hostnames matching
.*\\.example\\.com.
A Service of type ExternalName with externalName: api.example.com is admitted. A Service of type ExternalName with externalName: api.bad.com is denied by the hostname allow-list.
A later deny rule can override an earlier allow rule:
rules:
- enforce:
action: allow
services:
types:
- ExternalName
externalNames:
hostnames:
- exp: ".*\\.example\\.com"
- enforce:
action: deny
services:
externalNames:
hostnames:
- exact:
- blocked.example.com
Here, api.example.com is allowed, but blocked.example.com is denied because the later deny rule matches.
A later allow rule can override an earlier deny rule:
rules:
- enforce:
action: deny
services:
nodePorts:
ports:
- from: 30080
to: 30080
- namespaceSelector:
matchLabels:
allow-special-nodeport: "true"
enforce:
action: allow
services:
types:
- NodePort
nodePorts:
ports:
- from: 30080
to: 30080
In namespaces labeled allow-special-nodeport=true, a NodePort Service using 30080 is admitted because the namespace-specific allow rule matches later.
Service Rule Caveats
Service enforcement is intentionally explicit. Keep the following behavior in mind:
| Behavior | Explanation |
|---|
services.types is the type gate | Type-specific sections do not automatically grant the Service type. Include the Service type in services.types when an allow-list for Service types is active. |
| Type-specific constraints create allow-lists for their values | If loadBalancers.cidrs, externalNames.hostnames, or nodePorts.ports is configured with action: allow, non-matching values are denied. |
loadBalancers.cidrs requires explicit values | When CIDR constraints are configured, LoadBalancer Services must set spec.loadBalancerIP or spec.loadBalancerSourceRanges. |
nodePorts.ports requires explicit node ports | When port constraints are configured, NodePort Services and LoadBalancer Services with node port allocation enabled must set spec.ports[].nodePort. |
| LoadBalancer node port allocation matters | LoadBalancer Services are subject to NodePort range checks unless spec.allocateLoadBalancerNodePorts: false is set. |
| Audit does not allow | A matching audit rule emits events and warnings but does not satisfy an allow-list. |
| Last matching allow or deny wins | Later matching allow or deny rules override earlier matching allow or deny rules. |
| Negation applies to the whole matcher | negate: true inverts the result of both exact and exp. |
| Namespace selectors affect projected rules | Rules with namespaceSelector only apply to namespaces matching the selector. |
Complete Service Enforcement Example
The following example combines type enforcement, LoadBalancer CIDR restrictions, ExternalName hostname restrictions, NodePort range restrictions, audit rules, and namespace-specific exceptions:
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
...
rules:
- enforce:
action: allow
services:
types:
- ClusterIP
- NodePort
- LoadBalancer
- ExternalName
- enforce:
action: allow
services:
loadBalancers:
cidrs:
- 10.0.0.2/32
- 10.0.1.0/24
- enforce:
action: allow
services:
externalNames:
hostnames:
- exact:
- internal.git.com
- exp: ".*\\.example\\.com"
- enforce:
action: allow
services:
nodePorts:
ports:
- from: 30000
to: 30100
- from: 30500
to: 30500
- enforce:
action: deny
services:
nodePorts:
ports:
- from: 30090
to: 30090
- enforce:
action: deny
services:
loadBalancers:
cidrs:
- 10.0.66.0/24
- enforce:
action: audit
services:
externalNames:
hostnames:
- exp: "audit\\..*"
- namespaceSelector:
matchLabels:
environment: prod
enforce:
action: allow
services:
loadBalancers:
cidrs:
- 10.0.171.0/24
With this configuration:
ClusterIP, NodePort, LoadBalancer, and ExternalName Services are valid Service types.- LoadBalancer IPs must be contained in
10.0.0.2/32 or 10.0.1.0/24. - Namespaces labeled
environment=prod can also use LoadBalancer IPs in 10.0.171.0/24. - ExternalName hostnames must be
internal.git.com or match .*\\.example\\.com. - Explicit node ports must be in
30000-30100 or equal to 30500. - Node port
30090 is denied even though it is inside the broader allowed range. - ExternalName hostnames matching
audit\\..* emit audit events and warnings. - Audit matches do not allow values that fail the allow-list.
8.2 - Permissions
Configure policies and restrictions on a per-Tenant basis with Rules
Declare permission distribution rules for the selected namespaces.
As an administrator, you can define promotion rules. A promotion rule selects ServiceAccounts within a Tenant based on specified conditions and assigns them predefined ClusterRoles.
The selected ClusterRoles are then applied across all namespaces belonging to the Tenant, or a selected subset of namespaces, with the corresponding ServiceAccounts configured as subjects. This allows a ServiceAccount in one namespace to automatically receive equivalent permissions in other namespaces of the same Tenant.
This feature is particularly useful in scenarios involving Tenant Replications, where consistent permissions across namespaces are required.
---
apiVersion: capsule.clastix.io/v1beta2
kind: Tenant
metadata:
name: solar
spec:
...
rules:
- permissions:
promotions:
# Every promoted ServiceAccount receives this ClusterRole in all Namespaces of Tenant solar.
- clusterRoles:
- "configmap-replicator"
# Every promoted ServiceAccount with the matching labels receives this ClusterRole.
- clusterRoles:
- "secret-replicator"
selector:
matchLabels:
super: "account"
- namespaceSelector:
matchExpressions:
- key: env
operator: In
values: ["prod"]
permissions:
promotions:
# Promoted ServiceAccounts receive this ClusterRole only in namespaces matching env=prod.
- clusterRoles:
- "secret-replicator:prod"
Make sure the ClusterRoles exist. Otherwise, the corresponding Tenant reports a reconciliation error:
conditions:
- lastTransitionTime: "2026-02-16T23:08:59Z"
message: 'cannot sync rolebindings items: rolebindings.rbac.authorization.k8s.io
"tenant-replicator" not found'
If you run Capsule in Strict Mode, the controller must be allowed to grant the corresponding permissions to the ServiceAccount in all selected Namespaces. You can aggregate the same ClusterRoles to the controller:
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: configmap-replicator
labels:
projectcapsule.dev/aggregate-to-controller: "true"
rules:
- apiGroups: [""]
resources: ["configmaps"]
verbs: ["get", "create", "patch", "watch", "list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: secret-replicator
labels:
projectcapsule.dev/aggregate-to-controller: "true"
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "create", "patch", "watch", "list", "delete"]
As a Tenant Owner, Alice can promote ServiceAccounts by labeling them with projectcapsule.dev/promote=true. This feature must be enabled in the CapsuleConfiguration. If the feature is disabled, admission fails:
Error from server (Forbidden): admission webhook "serviceaccounts.projectcapsule.dev" denied the request: service account promotion is disabled. Contact cluster administrators
When the feature is enabled, the following command succeeds, assuming alice is a Tenant Owner of the solar Tenant:
kubectl label sa gitops-reconcile -n solar-test projectcapsule.dev/promote=true --as alice --as-group projectcapsule.dev
Verify the promotion in the Tenant status:
kubectl get tnt solar -o jsonpath='{.status.promotions}' | jq
Example status:
[
{
"clusterRoles": [
"tenant-replicator"
],
"kind": "ServiceAccount",
"name": "system:serviceaccount:solar-test:gitops-reconcile",
"targets": [
"solar-test",
"solar-prod"
]
}
]
You can verify that the RoleBinding was distributed to other namespaces of the solar Tenant:
kubectl get rolebinding -n solar-prod
NAME ROLE AGE
..
capsule:managed:7ad688b586eada40 ClusterRole/configmap-replicator 21s
..
To revoke the promotion, Alice can remove the label:
kubectl label sa gitops-reconcile -n solar-test projectcapsule.dev/promote- --as alice --as-group projectcapsule.dev