
Title
Create new category
Edit page index title
Edit category
Edit link
The tenant list/projects dropdown shows empty in UI post login
Problem
It was observed that the tenants/projects list shows up empty when the user logs in to a Region with a large number of tenants. This was consistent for SSO and local users. The UI inspect was showing a lot of calls failing with "net error":"ERR HTTP2 PROTOCOL ERROR"
Environment
- Private Cloud Director Virtualization - v2025.4 to v2025.8
- Private Cloud Director Kubernetes – v2025.4 to v2025.8
- Self-Hosted Private Cloud Director Virtualization - v2025.4 to v2025.8
- Self-Hosted Private Cloud Director Kubernetes - v2025.4 to v2025.8
- Component: UI
Cause
The cause of the problem was the nginx response headers being too large and being dropped or truncated on response during the login process. This is now tracked on PCD-3830, and the fix is planned for the October release.
Diagnostics
- It was noticed that the error was seen only for a user with access to a large number of tenants, which could impact the response header or response body size
- The issue was not reproduced for users with access to a lesser number of tenants.
- On browser, run an inspect on the page and check for
"net error":"ERR HTTP2 PROTOCOL ERROR" - Debug logs were enabled on the ingress-nginx-controller on the control plane side by adding
--v=5to the startup params in the deployment. The nginx debug logs are shown below, which confirmed the issue
[info] 71#71: *484 client sent too large header field while processing HTTP/2 connection, client: [IP_ADDRESS], server: 0.0.0.0:443 [info] 71#71: *568 client sent too large header field while processing HTTP/2 connection, client: [IP_ADDRESS], server: 0.0.0.0:443Step 4 can be performed on Self-Hosted PCD
Workaround
Until the fix is part of the product, follow the workaround.
The below workaround can be applied for Self-Hosted PCD. For SaaS customers, please reach out to the Platform9 Support Team.
The following changes were made to the ingress-nginx-controller deployment on the control plane, which fixed the problem.
a) The config map was edited to add buffer params client-header-buffer-size and large-client-header-buffers as below:
xxxxxxxxxx$ kubectl edit cm ingress-nginx-controller -n <WORKLOAD_NAMESPACE>apiVersion: v1data: allow-snippet-annotations: "true" annotations-risk-level: Critical client-header-buffer-size: 64k large-client-header-buffers: 4 32kkind: ConfigMapb) After making the changes the ingress-nginx-controller deployment was restarted
xxxxxxxxxx$ kubectl rollout restart deploy/ingress-nginx-controller -n <WORKLOAD_NAMESPACE>Validation
After applying the changes, the tenant dropdown list was verified as loading ok for various browsers and users.