If you're maintaining a CSI driver that relies on service account tokens, Kubernetes v1.35 introduces an important security refinement. Since the TokenRequests feature launched, service account tokens requested by CSI drivers have been delivered through the volume_context field. While functional, this approach has proven problematic for handling sensitive credentials, leading to accidental token exposure in CSI driver logs.
Kubernetes v1.35 addresses this with a beta feature: CSI Driver Opt-in for Service Account Tokens via Secrets Field. This enhancement enables CSI drivers to receive service account tokens through the secrets field in NodePublishVolumeRequest, aligning with the CSI specification's intended mechanism for sensitive data.
Understanding the existing approach
CSI drivers using the TokenRequests feature can request service account tokens for workload identity by configuring the TokenRequests field in their CSIDriver spec. These tokens are delivered as part of the volume attributes map under the key csi.storage.k8s.io/serviceAccount.tokens.
While volume_context works functionally, it wasn't designed for sensitive data, creating two significant problems:
The protosanitizer library used by CSI drivers doesn't treat volume context as sensitive, allowing service account tokens to leak into logs when gRPC requests are recorded. This vulnerability materialized in CVE-2023-2878 affecting the Secrets Store CSI Driver and CVE-2024-3744 in the Azure File CSI Driver.
Additionally, each CSI driver must implement custom sanitization logic to prevent token exposure, resulting in inconsistent security practices across the ecosystem.
The CSI specification already defines a secrets field in NodePublishVolumeRequest specifically for sensitive information. However, migrating tokens to this field requires careful handling to avoid breaking existing CSI drivers that expect tokens in volume context.
How the opt-in mechanism works
Kubernetes v1.35 introduces an opt-in mechanism allowing CSI drivers to specify how they receive service account tokens. This preserves backward compatibility while enabling drivers to adopt the more secure delivery method when ready.
CSI drivers configure this behavior through a new field in their CSIDriver spec:
#
# CAUTION: this is an example configuration.
# Do not use this for your own cluster!
#
apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
name: example-csi-driver
spec:
# ... existing fields ...
tokenRequests:
- audience: "example.com"
expirationSeconds: 3600
# New field for opting into secrets delivery
serviceAccountTokenInSecrets: true # defaults to false
The serviceAccountTokenInSecrets field controls token delivery:
When false (the default), tokens are placed in VolumeContext using the key csi.storage.k8s.io/serviceAccount.tokens, maintaining current behavior. When true, tokens are delivered exclusively through the Secrets field with the same key.
About the beta release
The CSIServiceAccountTokenSecrets feature gate is enabled by default on both kubelet and kube-apiserver. Since serviceAccountTokenInSecrets defaults to false, enabling the feature gate produces no behavioral changes. All drivers continue receiving tokens via volume context unless explicitly configured otherwise, which justified starting at beta rather than alpha.
Guide for CSI driver authors
If you maintain a CSI driver using service account tokens, here's how to adopt this feature.
Adding fallback logic
Update your driver code to check both token locations, ensuring compatibility with old and new delivery methods:
const serviceAccountTokenKey = "csi.storage.k8s.io/serviceAccount.tokens"
func getServiceAccountTokens(req *csi.NodePublishVolumeRequest) (string, error) {
// Check secrets field first (new behavior when driver opts in)
if tokens, ok := req.Secrets[serviceAccountTokenKey]; ok {
return tokens, nil
}
// Fall back to volume context (existing behavior)
if tokens, ok := req.VolumeContext[serviceAccountTokenKey]; ok {
return tokens, nil
}
return "", fmt.Errorf("service account tokens not found")
}
This fallback logic is backward compatible and safe to deploy in any driver version, even before clusters upgrade to v1.35.
Rollout sequence
CSI driver authors must follow a specific sequence to avoid breaking existing volumes.
Driver preparation (can happen anytime)
Add fallback logic that checks both the secrets field and volume context for tokens. This change is backward compatible and safe to ship immediately, even before clusters upgrade to v1.35. Consider adding this logic early, cutting releases, and backporting to maintenance branches where feasible.
Cluster upgrade and feature enablement
Once your driver includes fallback logic, follow this rollout order:
- Complete the kube-apiserver upgrade to 1.35 or later
- Complete kubelet upgrade to 1.35 or later on all nodes
- Ensure CSI driver version with fallback logic is deployed (if not already done in preparation phase)
- Fully complete CSI driver DaemonSet rollout across all nodes
- Update your CSIDriver manifest to set
serviceAccountTokenInSecrets: true
Important constraints
Timing is critical. If your CSI driver DaemonSet and CSIDriver object are in the same manifest or Helm chart, you need two separate updates. Deploy the new driver version with fallback logic first, wait for complete DaemonSet rollout, then update the CSIDriver spec to set serviceAccountTokenInSecrets: true.
Don't update the CSIDriver before all driver pods have rolled out. Doing so will cause volume mount failures on nodes still running the old driver version, since those pods only check volume context.
Why this matters
Adopting this feature provides several benefits:
- Eliminates the risk of accidentally logging service account tokens as part of volume context in gRPC requests
- Uses the CSI specification's designated field for sensitive data, following established best practices
- The
protosanitizertool automatically handles the secrets field correctly, eliminating the need for driver-specific workarounds - Opt-in design allows migration at your own pace without breaking existing deployments
Call to action
Kubernetes SIG Storage encourages CSI driver authors to adopt this feature and share feedback on the migration experience. If you have thoughts on the API design or encounter issues during adoption, reach out on the #csi channel on Kubernetes Slack (visit https://slack.k8s.io/ for an invitation).
Track progress across upcoming Kubernetes releases by following KEP-5538.
