ssvm: delete temp directory while deleting entity download url #12562
+131
−6
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
url When expired entity URLs are cleaned up, orphan directories are left behind. This change tries to delete the base temporary directory while cleanup. Signed-off-by: Abhishek Kumar <abhishek.mrt22@gmail.com>
14 tasks
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## 4.20 #12562 +/- ##
=========================================
Coverage 16.26% 16.26%
- Complexity 13426 13432 +6
=========================================
Files 5660 5660
Lines 499949 499982 +33
Branches 60704 60712 +8
=========================================
+ Hits 81296 81332 +36
+ Misses 409581 409572 -9
- Partials 9072 9078 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Contributor
Author
|
@blueorangutan package |
|
@shwstppr a [SL] Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress. |
|
Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16663 |
Contributor
Author
|
@blueorangutan test |
|
@shwstppr a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests |
|
[SF] Trillian test result (tid-15346)
|
shwstppr
changed the title
RosiKyu
suggested changes
Feb 4, 2026
Collaborator
RosiKyu
left a comment
RosiKyu
left a comment
There was a problem hiding this comment.
I have identified an issue in the PR:
Test Steps
- Configure short expiration intervals:
extract.url.expiration.interval= 120 secondsextract.url.cleanup.interval= 60 seconds
- Restart management server for config changes to take effect
- Create and attach a DATA volume to a VM
- Stop the VM
- Extract the volume to generate a download URL
- Verify UUID directory is created in
/var/www/html/userdata/ - Wait for URL expiration and cleanup (~3 minutes)
- Check if symlink and UUID directory are deleted
Expected Result:
- The symlink
/var/www/html/userdata/<UUID>/test-vol.qcow2should be deleted - The UUID directory
/var/www/html/userdata/<UUID>/should be deleted - SSVM logs should show: "Deleting symlink root directory: for "
Actual Result:
- FAIL - The unlink command failed due to incorrect path construction
- The unlink tried
/var/www/html/userdata/test-vol.qcow2instead of/var/www/html/userdata/7357f278-a8be-4c93-a43a-4679f6f220ab/test-vol.qcow2 - Since unlink failed, the
deleteEntitySymlinkRootPathIfNeededmethod never executed - Both the symlink AND UUID directory remain orphaned
Root Cause:
There seems to be an issue in the path extraction logic:
String linkPath = extractUrl.substring(extractUrl.lastIndexOf(File.separator) + 1);This extracts only the filename (test-vol.qcow2) instead of the full relative path (7357f278-a8be-4c93-a43a-4679f6f220ab/test-vol.qcow2).
Test Evidence:
- Configuration changes:
(localcloud) 🐱 > update configuration name=extract.url.expiration.interval value=120
{
"configuration": {
"category": "Advanced",
"component": "management-server",
"defaultvalue": "14400",
"description": "The life of an extract URL after which it is deleted ",
"displaytext": "Extract url expiration interval",
"group": "Management Server",
"isdynamic": false,
"name": "extract.url.expiration.interval",
"subgroup": "Limits",
"type": "Number",
"value": "120"
}
}
(localcloud) 🐱 > update configuration name=extract.url.cleanup.interval value=60
{
"configuration": {
"category": "Advanced",
"component": "management-server",
"defaultvalue": "7200",
"description": "The interval (in seconds) to wait before cleaning up the extract URL's ",
"displaytext": "Extract url cleanup interval",
"group": "Management Server",
"isdynamic": false,
"name": "extract.url.cleanup.interval",
"subgroup": "Limits",
"type": "Number",
"value": "60"
}
}
- Volume creation and attachment:
(localcloud) 🐱 > attach volume id=baff5910-2d21-4542-b2ee-6ee34cc6be73 virtualmachineid=90e7a447-1df2-4e75-8afc-ba807548ca42
{
"volume": {
"account": "admin",
"attached": "2026-02-04T11:45:14+0000",
"clusterid": "6f535ef8-be7f-4a5a-b87c-c589d427341b",
"clustername": "p1-c1",
"created": "2026-02-04T11:44:36+0000",
"deleteprotection": false,
"destroyed": false,
"deviceid": 1,
"diskioread": 0,
"diskiowrite": 0,
"diskkbsread": 0,
"diskkbswrite": 0,
"diskofferingdisplaytext": "Small Disk, 5 GB",
"diskofferingid": "4391aa82-7d9b-4e3c-9f16-9a4506f58c68",
"diskofferingname": "Small",
"displayvolume": true,
"domain": "ROOT",
"domainid": "ae560d02-01b6-11f1-9894-1e003100046f",
"domainpath": "/",
"hasannotations": false,
"hypervisor": "KVM",
"id": "baff5910-2d21-4542-b2ee-6ee34cc6be73",
"isextractable": true,
"jobid": "1c3936a5-d499-45de-a2d7-0495df953d8e",
"jobstatus": 0,
"name": "test-vol",
"path": "baff5910-2d21-4542-b2ee-6ee34cc6be73",
"podid": "09b0e9f6-5957-4f76-95ed-00764e23ed02",
"podname": "Pod1",
"provisioningtype": "thin",
"quiescevm": false,
"size": 5368709120,
"state": "Ready",
"storage": "ref-trl-10829-k-Mol9-rositsa-kyuchukova-kvm-pri2",
"storageid": "234e7aff-ac28-3150-8891-497563810b52",
"storagetype": "shared",
"supportsstoragesnapshot": false,
"tags": [],
"type": "DATADISK",
"virtualmachineid": "90e7a447-1df2-4e75-8afc-ba807548ca42",
"vmdisplayname": "VM-90e7a447-1df2-4e75-8afc-ba807548ca42",
"vmname": "VM-90e7a447-1df2-4e75-8afc-ba807548ca42",
"vmstate": "Running",
"vmtype": "User",
"zoneid": "7bc0e169-5133-42ba-8e5f-ea6f5db39ee8",
"zonename": "ref-trl-10829-k-Mol9-rositsa-kyuchukova"
}
}
- Volume extraction:
(localcloud) 🐱 > extract volume id=baff5910-2d21-4542-b2ee-6ee34cc6be73 mode=HTTP_DOWNLOAD zoneid=7bc0e169-5133-42ba-8e5f-ea6f5db39ee8
{
"volume": {
"accountid": "ec2bc4c5-01b6-11f1-9894-1e003100046f",
"extractMode": "HTTP_DOWNLOAD",
"id": "baff5910-2d21-4542-b2ee-6ee34cc6be73",
"name": "test-vol",
"state": "DOWNLOAD_URL_CREATED",
"url": "http://10.0.55.122/userdata/7357f278-a8be-4c93-a43a-4679f6f220ab/test-vol.qcow2",
"zoneid": "7bc0e169-5133-42ba-8e5f-ea6f5db39ee8",
"zonename": "ref-trl-10829-k-Mol9-rositsa-kyuchukova"
}
}
- UUID directory and symlink created:
root@s-3-VM:~# ls -la /var/www/html/userdata/
total 16
drwxr-xr-x 3 www-data www-data 4096 Feb 4 11:52 .
drwxr-xr-x 5 www-data www-data 4096 Feb 4 11:05 ..
drwxr-xr-x 2 www-data www-data 4096 Feb 4 11:52 7357f278-a8be-4c93-a43a-4679f6f220ab
-rwxr-xr-x 1 www-data www-data 17 Jan 31 10:47 .htaccess
root@s-3-VM:~# ls -la /var/www/html/userdata/7357f278-a8be-4c93-a43a-4679f6f220ab/
total 12
drwxr-xr-x 2 www-data www-data 4096 Feb 4 11:52 .
drwxr-xr-x 3 www-data www-data 4096 Feb 4 11:52 ..
lrwxrwxrwx 1 root root 107 Feb 4 11:52 test-vol.qcow2 -> /mnt/SecStorage/af60251b-b7ae-3132-a37a-9ba1f92e3118/volumes/2/5/3fb4a0d4-4d0d-489d-bb5f-77a85aa62fcc.qcow2
- Management Server log showing correct extractUrl sent:
[root@ref-trl-10829-k-Mol9-rositsa-kyuchukova-mgmt1 ~]# grep -i "DeleteEntityDownloadURL" /var/log/cloudstack/management/management-server.log | tail -20
2026-02-04 11:55:06,720 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (StorageManager-Scavenger-1:[]) (logid:) getCommandHostDelegation: class com.cloud.agent.api.storage.DeleteEntityDownloadURLCommand
2026-02-04 11:55:06,721 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (StorageManager-Scavenger-1:[]) (logid:) Wait time setting on com.cloud.agent.api.storage.DeleteEntityDownloadURLCommand is 1800 seconds
2026-02-04 11:55:06,722 DEBUG [c.c.a.t.Request] (StorageManager-Scavenger-1:[]) (logid:) Seq 4-4037477065937649683: Sending { Cmd , MgmtId: 32986170917999, via: 4(s-3-VM), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.storage.DeleteEntityDownloadURLCommand":{"path":"volumes/2/5/3fb4a0d4-4d0d-489d-bb5f-77a85aa62fcc.qcow2","extractUrl":"http://10.0.55.122/userdata/7357f278-a8be-4c93-a43a-4679f6f220ab/test-vol.qcow2","type":"VOLUME","parentPath":"af60251b-b7ae-3132-a37a-9ba1f92e3118","accountId":"0","wait":"0","bypassHostMaintenance":"false"}}] }
- SSVM log showing unlink failure due to incorrect path:
root@s-3-VM:~# grep -A5 -B5 "handleDeleteEntityDownloadURLCommand" /var/log/cloud.log
2026-02-04T11:55:06,734 WARN [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) handleDeleteEntityDownloadURLCommand Path:volumes/2/5/3fb4a0d4-4d0d-489d-bb5f-77a85aa62fcc.qcow2 Type:VOLUME
2026-02-04T11:55:06,756 WARN [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Execution of process [4146] for command [/bin/bash -c unlink /var/www/html/userdata/test-vol.qcow2 ] failed.
2026-02-04T11:55:06,757 WARN [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Process [4146] for command [/bin/bash -c unlink /var/www/html/userdata/test-vol.qcow2 ] encountered the error: [unlink: cannot unlink '/var/www/html/userdata/test-vol.qcow2': No such file or directory].
2026-02-04T11:55:06,757 WARN [storage.template.UploadManagerImpl] (AgentRequest-Handler-5:[]) Error in deleting symlink :unlink: cannot unlink '/var/www/html/userdata/test-vol.qcow2': No such file or directory
- After cleanup interval - UUID directory still exists (orphaned):
root@s-3-VM:~# ls -la /var/www/html/userdata/
total 16
drwxr-xr-x 3 www-data www-data 4096 Feb 4 11:52 .
drwxr-xr-x 5 www-data www-data 4096 Feb 4 11:05 ..
drwxr-xr-x 2 www-data www-data 4096 Feb 4 11:52 7357f278-a8be-4c93-a43a-4679f6f220ab
-rwxr-xr-x 1 www-data www-data 17 Jan 31 10:47 .htaccess
- Verified PR code is deployed (method exists in JAR):
root@s-3-VM:/usr/local/cloud/systemvm# unzip -p cloud-secondary-storage-4.20.3.0-SNAPSHOT.jar | grep -a "deleteEntitySymlinkRootPathIfNeeded"
extractUrlkl#deleteEntitySymlinkRootPathIfNeededQ(Lcom/cloud/agent/api/storage/DeleteEntityDownloadURLCommand;Ljava/lang/String;)V...
root@s-3-VM:/usr/local/cloud/systemvm# unzip -p cloud-secondary-storage-4.20.3.0-SNAPSHOT.jar | grep -a "Deleting symlink root directory"
...Deleting symlink root directory: {} for {}...
The path extraction logic needs to be fixed. The code should extract everything after /userdata/ from the extractUrl, not just the filename after the last /.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
When expired entity URLs are cleaned up, orphan directories are left behind. This change tries to delete the base temporary directory while cleanup.
Types of changes
Feature/Enhancement Scale or Bug Severity
Feature/Enhancement Scale
Bug Severity
Screenshots (if appropriate):
How Has This Been Tested?
Download link for a volume expires and server calls DeleteEntityDownloadURLCommand
Before
After
How did you try to break this feature and the system with this change?