Debugging Kubernetes control plane components presents significant challenges, particularly when you need rapid insight into runtime state or configuration verification. Kubernetes 1.35 addresses this by enhancing z-pages debugging endpoints with structured, machine-parseable responses that streamline tooling development and automate troubleshooting workflows.
What are z-pages?
z-pages are specialized debugging endpoints exposed by Kubernetes control plane components. First introduced as an alpha feature in Kubernetes 1.32, these endpoints deliver runtime diagnostics for kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, and kube-proxy. The "z-pages" naming convention derives from the /*z path pattern used for debugging endpoints.
Kubernetes currently supports two core z-page endpoints:
/statusz- Provides high-level component information including version details, start time, uptime, and available debug paths
/flagz- Exposes all command-line arguments and their values used to launch the component, with sensitive values automatically redacted for security
While these endpoints prove valuable for operators performing quick component inspections, they previously returned only plain text output that resisted programmatic parsing.
What's new in Kubernetes 1.35?
Kubernetes 1.35 introduces structured, versioned responses for both /statusz and /flagz endpoints. This enhancement preserves backward compatibility with existing plain text formats while adding machine-readable JSON response support.
Backward compatible design
The structured responses operate on an opt-in basis. Without an Accept header, endpoints continue returning the familiar plain text format:
$ curl --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
--key /etc/kubernetes/pki/apiserver-kubelet-client.key \
--cacert /etc/kubernetes/pki/ca.crt \
https://localhost:6443/statusz
kube-apiserver statusz
Warning: This endpoint is not meant to be machine parseable, has no formatting compatibility guarantees and is for debugging purposes only.
Started: Wed Oct 16 21:03:43 UTC 2024
Up: 0 hr 00 min 16 sec
Go version: go1.23.2
Binary version: 1.35.0-alpha.0.1595
Emulation version: 1.35
Paths: /healthz /livez /metrics /readyz /statusz /version
Structured JSON responses
To receive a structured response, include the appropriate Accept header:
Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Statusz
This returns a versioned JSON response:
{
"kind": "Statusz",
"apiVersion": "config.k8s.io/v1alpha1",
"metadata": {
"name": "kube-apiserver"
},
"startTime": "2025-10-29T00:30:01Z",
"uptimeSeconds": 856,
"goVersion": "go1.23.2",
"binaryVersion": "1.35.0",
"emulationVersion": "1.35",
"paths": [
"/healthz",
"/livez",
"/metrics",
"/readyz",
"/statusz",
"/version"
]
}
The /flagz endpoint supports structured responses with this header:
Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Flagz
Example response:
{
"kind": "Flagz",
"apiVersion": "config.k8s.io/v1alpha1",
"metadata": {
"name": "kube-apiserver"
},
"flags": {
"advertise-address": "192.168.8.4",
"allow-privileged": "true",
"authorization-mode": "[Node,RBAC]",
"enable-priority-and-fairness": "true",
"profiling": "true"
}
}
Why structured responses matter
Structured responses unlock several practical capabilities:
1. Automated health checks and monitoring
Monitoring tools can now extract specific fields directly without text parsing. You can programmatically verify that components aren't running with unexpected emulated versions or confirm that critical flags are properly configured.
2. Better debugging tools
Developers can build sophisticated debugging utilities that compare configurations across multiple components or track configuration drift over time. The structured format enables straightforward configuration diffs and validation that components run with expected settings.
3. API versioning and stability
Versioned APIs (beginning with v1alpha1) establish a clear path toward stability. As the feature matures through v1beta1 to eventual v1 status, your tooling gains protection against breaking changes in future Kubernetes releases.
How to use structured z-pages
Prerequisites
Both endpoints require feature gate enablement:
/statusz: Enable theComponentStatuszfeature gate/flagz: Enable theComponentFlagzfeature gate
Example: Getting structured responses
Here's how to retrieve structured JSON responses from the kube-apiserver using curl:
# Retrieve structured statusz response
curl \
--cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
--key /etc/kubernetes/pki/apiserver-kubelet-client.key \
--cacert /etc/kubernetes/pki/ca.crt \
-H "Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Statusz" \
https://localhost:6443/statusz | jq .
# Retrieve structured flagz response
curl \
--cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
--key /etc/kubernetes/pki/apiserver-kubelet-client.key \
--cacert /etc/kubernetes/pki/ca.crt \
-H "Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Flagz" \
https://localhost:6443/flagz | jq .
Note:
These examples authenticate using client certificates and validate the server certificate with--cacert. In test environments, you can bypass certificate verification using --insecure or -k, though this practice exposes you to man-in-the-middle attacks and should never be used in production.Important considerations
Alpha feature status
Structured z-page responses remain an alpha feature in Kubernetes 1.35, which carries several implications:
- The API schema may evolve or break between releases
- These endpoints serve debugging purposes rather than production automation
- Avoid building critical monitoring workflows around them until they achieve beta or stable status
Security and access control
z-pages expose sensitive component internals and demand careful access management. Consider these security factors:
Authorization: Only members of the system:monitoring group can access z-page endpoints, following the same authorization model as debugging endpoints like /healthz, /livez, and /readyz. This restriction ensures debugging information remains available only to authorized users and service accounts. In RBAC-enabled clusters, manage access by assigning appropriate permissions to this group.
Authentication: Authentication requirements vary based on cluster configuration. Most clusters require authentication mechanisms such as client certificates unless anonymous authentication is explicitly enabled.
Information disclosure: These endpoints expose detailed cluster component information, including:
- Component versions and build metadata
- Complete command-line arguments and values (with sensitive data redacted)
- Available debug endpoints
Limit access to trusted operators and debugging tools. Never expose these endpoints to unauthorized users or automated systems lacking legitimate debugging requirements.
Future evolution
As this feature matures, Kubernetes SIG Instrumentation plans to:
- Release
v1beta1and eventuallyv1API versions - Incorporate community feedback into the response schema
- Expand z-page endpoints based on user requirements
Try it out
Test structured z-pages in a non-production environment:
- Enable the
ComponentStatuszandComponentFlagzfeature gates on control plane components - Query endpoints using both plain text and structured formats
- Build a simple script or tool that consumes the structured data
- Provide feedback to the community
Learn more
- z-pages documentation
- KEP-4827: Component Statusz
- KEP-4828: Component Flagz
- Join the discussion in the #sig-instrumentation channel on Kubernetes Slack
Get involved
SIG Instrumentation welcomes your feedback on structured z-pages. This feature aims to simplify Kubernetes debugging and monitoring, and your experience—whether building internal tools, contributing to open source projects, or exploring the functionality—directly influences Kubernetes observability development.
For questions, suggestions, or issues, contact SIG Instrumentation through Slack or attend our regular community meetings.
Happy debugging!