Investing

Kubernetes 1.35 Brings Versioned z-pages APIs for Streamlined Debugging Workflows

2025-12-31 10:30
332 views

Debugging Kubernetes control plane components presents significant challenges, particularly when you need rapid insight into runtime state or configuration verification. Kubernetes 1.35 addresses this by enhancing z-pages debugging endpoints with structured, machine-parseable responses that streamline tooling development and automate troubleshooting workflows.

What are z-pages?

z-pages are specialized debugging endpoints exposed by Kubernetes control plane components. First introduced as an alpha feature in Kubernetes 1.32, these endpoints deliver runtime diagnostics for kube-apiserver, kube-controller-manager, kube-scheduler, kubelet, and kube-proxy. The "z-pages" naming convention derives from the /*z path pattern used for debugging endpoints.

Kubernetes currently supports two core z-page endpoints:

/statusz
Provides high-level component information including version details, start time, uptime, and available debug paths
/flagz
Exposes all command-line arguments and their values used to launch the component, with sensitive values automatically redacted for security

While these endpoints prove valuable for operators performing quick component inspections, they previously returned only plain text output that resisted programmatic parsing.

What's new in Kubernetes 1.35?

Kubernetes 1.35 introduces structured, versioned responses for both /statusz and /flagz endpoints. This enhancement preserves backward compatibility with existing plain text formats while adding machine-readable JSON response support.

Backward compatible design

The structured responses operate on an opt-in basis. Without an Accept header, endpoints continue returning the familiar plain text format:

$ curl --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
--key /etc/kubernetes/pki/apiserver-kubelet-client.key \
--cacert /etc/kubernetes/pki/ca.crt \
https://localhost:6443/statusz
kube-apiserver statusz
Warning: This endpoint is not meant to be machine parseable, has no formatting compatibility guarantees and is for debugging purposes only.
Started: Wed Oct 16 21:03:43 UTC 2024
Up: 0 hr 00 min 16 sec
Go version: go1.23.2
Binary version: 1.35.0-alpha.0.1595
Emulation version: 1.35
Paths: /healthz /livez /metrics /readyz /statusz /version

Structured JSON responses

To receive a structured response, include the appropriate Accept header:

Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Statusz

This returns a versioned JSON response:

{
 "kind": "Statusz",
 "apiVersion": "config.k8s.io/v1alpha1",
 "metadata": {
 "name": "kube-apiserver"
 },
 "startTime": "2025-10-29T00:30:01Z",
 "uptimeSeconds": 856,
 "goVersion": "go1.23.2",
 "binaryVersion": "1.35.0",
 "emulationVersion": "1.35",
 "paths": [
 "/healthz",
 "/livez",
 "/metrics",
 "/readyz",
 "/statusz",
 "/version"
 ]
}

The /flagz endpoint supports structured responses with this header:

Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Flagz

Example response:

{
 "kind": "Flagz",
 "apiVersion": "config.k8s.io/v1alpha1",
 "metadata": {
 "name": "kube-apiserver"
 },
 "flags": {
 "advertise-address": "192.168.8.4",
 "allow-privileged": "true",
 "authorization-mode": "[Node,RBAC]",
 "enable-priority-and-fairness": "true",
 "profiling": "true"
 }
}

Why structured responses matter

Structured responses unlock several practical capabilities:

1. Automated health checks and monitoring

Monitoring tools can now extract specific fields directly without text parsing. You can programmatically verify that components aren't running with unexpected emulated versions or confirm that critical flags are properly configured.

2. Better debugging tools

Developers can build sophisticated debugging utilities that compare configurations across multiple components or track configuration drift over time. The structured format enables straightforward configuration diffs and validation that components run with expected settings.

3. API versioning and stability

Versioned APIs (beginning with v1alpha1) establish a clear path toward stability. As the feature matures through v1beta1 to eventual v1 status, your tooling gains protection against breaking changes in future Kubernetes releases.

How to use structured z-pages

Prerequisites

Both endpoints require feature gate enablement:

  • /statusz: Enable the ComponentStatusz feature gate
  • /flagz: Enable the ComponentFlagz feature gate

Example: Getting structured responses

Here's how to retrieve structured JSON responses from the kube-apiserver using curl:

# Retrieve structured statusz response
curl \
 --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
 --key /etc/kubernetes/pki/apiserver-kubelet-client.key \
 --cacert /etc/kubernetes/pki/ca.crt \
 -H "Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Statusz" \
 https://localhost:6443/statusz | jq .

# Retrieve structured flagz response
curl \
 --cert /etc/kubernetes/pki/apiserver-kubelet-client.crt \
 --key /etc/kubernetes/pki/apiserver-kubelet-client.key \
 --cacert /etc/kubernetes/pki/ca.crt \
 -H "Accept: application/json;v=v1alpha1;g=config.k8s.io;as=Flagz" \
 https://localhost:6443/flagz | jq .

Note:

These examples authenticate using client certificates and validate the server certificate with --cacert. In test environments, you can bypass certificate verification using --insecure or -k, though this practice exposes you to man-in-the-middle attacks and should never be used in production.

Important considerations

Alpha feature status

Structured z-page responses remain an alpha feature in Kubernetes 1.35, which carries several implications:

  • The API schema may evolve or break between releases
  • These endpoints serve debugging purposes rather than production automation
  • Avoid building critical monitoring workflows around them until they achieve beta or stable status

Security and access control

z-pages expose sensitive component internals and demand careful access management. Consider these security factors:

Authorization: Only members of the system:monitoring group can access z-page endpoints, following the same authorization model as debugging endpoints like /healthz, /livez, and /readyz. This restriction ensures debugging information remains available only to authorized users and service accounts. In RBAC-enabled clusters, manage access by assigning appropriate permissions to this group.

Authentication: Authentication requirements vary based on cluster configuration. Most clusters require authentication mechanisms such as client certificates unless anonymous authentication is explicitly enabled.

Information disclosure: These endpoints expose detailed cluster component information, including:

  • Component versions and build metadata
  • Complete command-line arguments and values (with sensitive data redacted)
  • Available debug endpoints

Limit access to trusted operators and debugging tools. Never expose these endpoints to unauthorized users or automated systems lacking legitimate debugging requirements.

Future evolution

As this feature matures, Kubernetes SIG Instrumentation plans to:

  • Release v1beta1 and eventually v1 API versions
  • Incorporate community feedback into the response schema
  • Expand z-page endpoints based on user requirements

Try it out

Test structured z-pages in a non-production environment:

  1. Enable the ComponentStatusz and ComponentFlagz feature gates on control plane components
  2. Query endpoints using both plain text and structured formats
  3. Build a simple script or tool that consumes the structured data
  4. Provide feedback to the community

Learn more

Get involved

SIG Instrumentation welcomes your feedback on structured z-pages. This feature aims to simplify Kubernetes debugging and monitoring, and your experience—whether building internal tools, contributing to open source projects, or exploring the functionality—directly influences Kubernetes observability development.

For questions, suggestions, or issues, contact SIG Instrumentation through Slack or attend our regular community meetings.

Happy debugging!