Image Download Fails With 504 Gateway Timeout

Problem

When attempting to download an image from Image service, the operation fails with a 504 Gateway Time-out error.

$ openstack image save --os-interface admin --insecure --file <FILENAME> <IMAGE_ID> HttpException: 504: Server Error for url: https://[IMAGE_LIBRARY_HOST_IP]:9494/v2/images/[IMAGE_ID]: 504 Gateway Time-out: nginx/1.18.0 (Ubuntu)

Environment

  • Image Service

  • Private Cloud Director Virtualization - v2025.4 and Higher

  • Self-Hosted Private Cloud Director Virtualization - v2025.4 and Higher

Cause

The active Image Library node was not using the shared NFS mount for the image directory; instead, it was using local storage.

As a result, when an image stored on the shared NFS was requested via the API endpoint on the image library node, failed to retrieve the image, leading to the 504 timeout error.

Diagnostics

  1. Verify the image status and ID:

$ openstack image list | grep <IMAGE_ID> | [IMAGE_ID]| [IMAGE_NAME] | active |

  1. Check the Image service endpoint configuration:

$ openstack endpoint list --service glance --region <REGION_NAME> | ID | Region | Service Name | Service Type | Enabled | Interface | URL | [ID] | [REGION_NAME] | glance | image | True | admin | [IMAGE_ENPOINT_URL]

Confirm that the IP-Address mentioned in the IMAGE-ENDPOINT-URL should belong to the correct Image Library node

  1. Confirm if the image file exists:

$ ls -l /var/lib/glance/images | grep <IMAGE_ID>

Image not found locally. However, the image found on another Image Library node.

  1. Check the mount status of the "images" directory on the Image node:

$ df -hT /var/lib/glance/images/

/etc/fstab has no NFS entry for the image directory. Whereas the other Image Library node has the NFS entry.

Info

The below glance-api log is visible only to Self-Hosted PCD-V environments. For SaaS environments, reach out to the Platform9 support team for further assistance.

  1. Log confirmation from glance-apilog:

Glance tried all active locations to get data for image [IMAGE_ID], but all have failed.

Resolution

Perform the following steps on the Image Library node:

NOTE

New image creation will be halted during this procedure. Perform during a maintenance window.

  1. Stop Image Library API service to prevent new image operations:

$ sudo systemctl stop pf9-glance-api

  1. Backup the existing local image directory:

$ sudo mv /var/lib/glance/images /var/lib/glance/images_old

  1. Create a new directory for NFS mount and set correct permissions:

$ sudo mkdir /var/lib/glance/images $ sudo chown pf9:pf9group /var/lib/glance/images

  1. Update /etc/fstab with the correct NFS share entry:

$ echo "<NFS_SERVER>:/<SHARE_PATH>/images /var/lib/glance/images nfs vers=3,proto=tcp,lookupcache=pos,noacl,nolock 0 0" | sudo tee -a /etc/fstab

  1. Mount all file systems:

$ sudo mount -a

  1. Verify that the mount is active:

$ df -hT /var/lib/glance/images
Filesystem Type Size Used Avail Use% Mounted on [NFS_SERVER]:/images nfs [SIZE] [SIZE] [SIZE] [USED_SIZE]% /var/lib/glance/images

  1. Sync old local images (if any) to the NFS mount:

$ sudo rsync -avP /var/lib/glance/images_old/ /var/lib/glance/images/

  1. Restart the Image Library service:

$ sudo systemctl start pf9-glance-api

  1. Validate image access:

Attempt to download the image again:

$ openstack image save --file <FILE_NAME> <IMAGE_ID>

  1. (Optional) After validation, remove the old directory to free up space:

$ sudo rm -rf /var/lib/glance/images_old

Additional Information

  • Ensure all Image Library nodes are configured consistently with the same NFS mount points for /var/lib/glance/images and /opt/data/instances.

  • This issue commonly occurs after a node rebuild or reconfiguration, where /etc/fstab entries may have been lost or skipped.


Next to read:
VM Creation Fails With ImageUnacceptable Error
null
On This Page
Image Download Fails With 504 Gateway TimeoutProblemEnvironmentCauseDiagnosticsResolutionAdditional Information