2016-11-15 16:29:19,037 DEBUG [c.c.a.ApiServlet] (catalina-exec-24:ctx-33bb8123) (logid:a4c2eb47) ===START=== 172.16.192.198 -- GET command=createSnapshot&response=json&volumeId=f4a5f87e-232b-4956-af21-a3bc232cf47b&quiescevm=false&name=test15nov&_=1479227442441 2016-11-15 16:29:19,103 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-24:ctx-8acb10b8 job-1375) (logid:7410162b) Add job-1375 into job monitoring 2016-11-15 16:29:19,106 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-24:ctx-33bb8123 ctx-c9fae40d) (logid:a4c2eb47) submit async job-1375, details: AsyncJobVO {id:1375, userId: 2, accountId: 2, instanceType: Snapshot, instanceId: 171, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo: {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"2","uuid":"1218306f-333c-45d2-96fa-5826d4c4d03a","cmdEventType":"SNAPSHOT.CREATE","response":"json","ctxUserId":"2","volumeId":"f4a5f87e-232b-4956-af21-a3bc232cf47b","name":"test15nov","ctxStartEventId":"3926","id":"171","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"f4a5f87e-232b-4956-af21-a3bc232cf47b\",\"interface com.cloud.storage.Snapshot\":\"1218306f-333c-45d2-96fa-5826d4c4d03a\"}","_":"1479227442441"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 14038011095064, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2016-11-15 16:29:19,107 DEBUG [c.c.a.ApiServlet] (catalina-exec-24:ctx-33bb8123 ctx-c9fae40d) (logid:a4c2eb47) ===END=== 172.16.192.198 -- GET command=createSnapshot&response=json&volumeId=f4a5f87e-232b-4956-af21-a3bc232cf47b&quiescevm=false&name=test15nov&_=1479227442441 2016-11-15 16:29:19,109 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:ctx-8acb10b8 job-1375) (logid:bd1d8a8f) Executing AsyncJobVO {id:1375, userId: 2, accountId: 2, instanceType: Snapshot, instanceId: 171, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo: {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"2","uuid":"1218306f-333c-45d2-96fa-5826d4c4d03a","cmdEventType":"SNAPSHOT.CREATE","response":"json","ctxUserId":"2","volumeId":"f4a5f87e-232b-4956-af21-a3bc232cf47b","name":"test15nov","ctxStartEventId":"3926","id":"171","ctxDetails":"{\"interface com.cloud.storage.Volume\":\"f4a5f87e-232b-4956-af21-a3bc232cf47b\",\"interface com.cloud.storage.Snapshot\":\"1218306f-333c-45d2-96fa-5826d4c4d03a\"}","_":"1479227442441"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 14038011095064, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2016-11-15 16:29:19,135 INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-24:ctx-8acb10b8 job-1375 ctx-0d61f0a4) (logid:bd1d8a8f) VOLSS: createSnapshotCmd starts:1479227359135 2016-11-15 16:29:19,153 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:ctx-8acb10b8 job-1375 ctx-0d61f0a4) (logid:bd1d8a8f) Sync job-1376 execution on object VmWorkJobQueue.68 2016-11-15 16:29:20,064 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-977298f7) (logid:9e22aa50) Execute sync-queue item: SyncQueueItemVO {id:25, queueId: 351, contentType: AsyncJob, contentId: 1376, lastProcessMsid: 14038011095064, lastprocessNumber: 166, lastProcessTime: Tue Nov 15 16:29:20 GMT 2016, created: Tue Nov 15 16:29:19 GMT 2016} 2016-11-15 16:29:20,065 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-977298f7) (logid:9e22aa50) Schedule queued job-1376 2016-11-15 16:29:20,080 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:b6824d86) Add job-1376 into job monitoring 2016-11-15 16:29:20,089 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Executing AsyncJobVO {id:1376, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAARaAAlxdWllc2NlVm1MAAhwb2xpY3lJZHQAEExqYXZhL2xhbmcvTG9uZztMAApzbmFwc2hvdElkcQB-AAFMAAh2b2x1bWVJZHEAfgABeHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAQAAAAAAAAAAgAAAAAAAABEdAAUVm9sdW1lQXBpU2VydmljZUltcGwAc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAAAc3EAfgAGAAAAAAAAAKtzcQB-AAYAAAAAAAAASA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 14038011095064, completeMsid: null, lastUpdated: null, lastPolled: null, created: Tue Nov 15 16:29:19 GMT 2016} 2016-11-15 16:29:20,090 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 68, job origin: 1375 2016-11-15 16:29:20,091 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Execute VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":72,"policyId":0,"snapshotId":171,"quiesceVm":false,"userId":2,"accountId":4,"vmId":68,"handlerName":"VolumeApiServiceImpl"} 2016-11-15 16:29:20,130 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CreateObjectCommand 2016-11-15 16:29:20,130 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CreateObjectCommand 2016-11-15 16:29:20,132 DEBUG [c.c.a.t.Request] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Seq 1-7805019629211344943: Sending { Cmd , MgmtId: 14038011095064, via: 1(xenserver-2917), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"f4a5f87e-232b-4956-af21-a3bc232cf47b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"64d28c3f-4953-3f6c-b8f7-5858029226d7","id":3,"poolType":"NetworkFilesystem","host":"192.168.200.78","path":"/ssd-stor/primary","port":2049,"url":"NetworkFilesystem://192.168.200.78/ssd-stor/primary/?ROLE=Primary&STOREUUID=64d28c3f-4953-3f6c-b8f7-5858029226d7"}},"name":"ROOT-68","size":214748364800,"path":"6914e6e6-8529-4a19-b4cf-efd620efce39","volumeId":72,"vmName":"i-4-68-VM","accountId":4,"format":"VHD","provisioningType":"THIN","id":72,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"c827e745-4209-4615-802b-9759e94a588e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"64d28c3f-4953-3f6c-b8f7-5858029226d7","id":3,"poolType":"NetworkFilesystem","host":"192.168.200.78","path":"/ssd-stor/primary","port":2049,"url":"NetworkFilesystem://192.168.200.78/ssd-stor/primary/?ROLE=Primary&STOREUUID=64d28c3f-4953-3f6c-b8f7-5858029226d7"}},"vmName":"i-4-68-VM","name":"test15nov","hypervisorType":"XenServer","id":171,"quiescevm":false,"parents":["c827e745-4209-4615-802b-9759e94a588e"],"physicalSize":0}},"wait":0}}] } 2016-11-15 16:29:20,132 DEBUG [c.c.a.t.Request] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Seq 1-7805019629211344943: Executing: { Cmd , MgmtId: 14038011095064, via: 1(xenserver-2917), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"f4a5f87e-232b-4956-af21-a3bc232cf47b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"64d28c3f-4953-3f6c-b8f7-5858029226d7","id":3,"poolType":"NetworkFilesystem","host":"192.168.200.78","path":"/ssd-stor/primary","port":2049,"url":"NetworkFilesystem://192.168.200.78/ssd-stor/primary/?ROLE=Primary&STOREUUID=64d28c3f-4953-3f6c-b8f7-5858029226d7"}},"name":"ROOT-68","size":214748364800,"path":"6914e6e6-8529-4a19-b4cf-efd620efce39","volumeId":72,"vmName":"i-4-68-VM","accountId":4,"format":"VHD","provisioningType":"THIN","id":72,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"c827e745-4209-4615-802b-9759e94a588e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"64d28c3f-4953-3f6c-b8f7-5858029226d7","id":3,"poolType":"NetworkFilesystem","host":"192.168.200.78","path":"/ssd-stor/primary","port":2049,"url":"NetworkFilesystem://192.168.200.78/ssd-stor/primary/?ROLE=Primary&STOREUUID=64d28c3f-4953-3f6c-b8f7-5858029226d7"}},"vmName":"i-4-68-VM","name":"test15nov","hypervisorType":"XenServer","id":171,"quiescevm":false,"parents":["c827e745-4209-4615-802b-9759e94a588e"],"physicalSize":0}},"wait":0}}] } 2016-11-15 16:29:20,132 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-368:ctx-196f2658) (logid:39747749) Seq 1-7805019629211344943: Executing request 2016-11-15 16:29:20,269 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) (logid:) SeqA 10-1096875: Processing Seq 10-1096875: { Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":42,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2016-11-15 16:29:20,271 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-7:null) (logid:) SeqA 10-1096875: Sending Seq 10-1096875: { Ans: , MgmtId: 14038011095064, via: 10, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2016-11-15 16:29:20,888 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-368:ctx-196f2658) (logid:bd1d8a8f) Seq 1-7805019629211344943: Response Received: 2016-11-15 16:29:20,888 DEBUG [c.c.a.t.Request] (DirectAgent-368:ctx-196f2658) (logid:bd1d8a8f) Seq 1-7805019629211344943: Processing: { Ans: , MgmtId: 14038011095064, via: 1(xenserver-2917), Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"51dc10b0-1aa3-4843-9177-6e6bb838f29f","id":0,"quiescevm":false,"physicalSize":0}},"result":true,"wait":0}}] } 2016-11-15 16:29:20,888 DEBUG [c.c.a.t.Request] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Seq 1-7805019629211344943: Received: { Ans: , MgmtId: 14038011095064, via: 1(xenserver-2917), Ver: v1, Flags: 10, { CreateObjectAnswer } } 2016-11-15 16:29:20,931 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT 2016-11-15 16:29:21,003 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand 2016-11-15 16:29:21,003 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand 2016-11-15 16:29:21,004 DEBUG [c.c.a.t.Request] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Seq 1-7805019629211344944: Sending { Cmd , MgmtId: 14038011095064, via: 1(xenserver-2917), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"51dc10b0-1aa3-4843-9177-6e6bb838f29f","volume":{"uuid":"f4a5f87e-232b-4956-af21-a3bc232cf47b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"64d28c3f-4953-3f6c-b8f7-5858029226d7","id":3,"poolType":"NetworkFilesystem","host":"192.168.200.78","path":"/ssd-stor/primary","port":2049,"url":"NetworkFilesystem://192.168.200.78/ssd-stor/primary/?ROLE=Primary&STOREUUID=64d28c3f-4953-3f6c-b8f7-5858029226d7"}},"name":"ROOT-68","size":214748364800,"path":"6914e6e6-8529-4a19-b4cf-efd620efce39","volumeId":72,"vmName":"i-4-68-VM","accountId":4,"format":"VHD","provisioningType":"THIN","id":72,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"c827e745-4209-4615-802b-9759e94a588e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"64d28c3f-4953-3f6c-b8f7-5858029226d7","id":3,"poolType":"NetworkFilesystem","host":"192.168.200.78","path":"/ssd-stor/primary","port":2049,"url":"NetworkFilesystem://192.168.200.78/ssd-stor/primary/?ROLE=Primary&STOREUUID=64d28c3f-4953-3f6c-b8f7-5858029226d7"}},"vmName":"i-4-68-VM","name":"test15nov","hypervisorType":"XenServer","id":171,"quiescevm":false,"parents":["c827e745-4209-4615-802b-9759e94a588e"],"physicalSize":0}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/72","volume":{"uuid":"f4a5f87e-232b-4956-af21-a3bc232cf47b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"64d28c3f-4953-3f6c-b8f7-5858029226d7","id":3,"poolType":"NetworkFilesystem","host":"192.168.200.78","path":"/ssd-stor/primary","port":2049,"url":"NetworkFilesystem://192.168.200.78/ssd-stor/primary/?ROLE=Primary&STOREUUID=64d28c3f-4953-3f6c-b8f7-5858029226d7"}},"name":"ROOT-68","size":214748364800,"path":"6914e6e6-8529-4a19-b4cf-efd620efce39","volumeId":72,"vmName":"i-4-68-VM","accountId":4,"format":"VHD","provisioningType":"THIN","id":72,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/4/72/eaa25c94-a322-4823-9a64-8b39dd6d2736","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.234.78/sata-stor/secondary","_role":"Image"}},"vmName":"i-4-68-VM","name":"test15nov","hypervisorType":"XenServer","id":171,"quiescevm":false,"parents":["snapshots/4/72/afca7b2b-2877-44bf-9a0a-6356965d0360","snapshots/4/72/60a2388b-ef18-487a-a87d-8c73ddbb1498","snapshots/4/72/85f183ba-b433-4c99-a0f7-c486c3c143a4","snapshots/4/72/fcb5445d-1a1e-4eae-a6c2-58dfa7969948","snapshots/4/72/e7bbceac-5e30-4201-a880-e170e0756264","snapshots/4/72/149d9cc8-821f-485c-8781-c4e411d112bb","snapshots/4/72/95977384-ccdc-4472-8e69-ad2610a6fd78","snapshots/4/72/eaa25c94-a322-4823-9a64-8b39dd6d2736"],"physicalSize":0}},"executeInSequence":true,"options":{"fullSnapshot":"false"},"options2":{},"wait":21600}}] } 2016-11-15 16:29:21,004 DEBUG [c.c.a.t.Request] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Seq 1-7805019629211344944: Executing: { Cmd , MgmtId: 14038011095064, via: 1(xenserver-2917), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"51dc10b0-1aa3-4843-9177-6e6bb838f29f","volume":{"uuid":"f4a5f87e-232b-4956-af21-a3bc232cf47b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"64d28c3f-4953-3f6c-b8f7-5858029226d7","id":3,"poolType":"NetworkFilesystem","host":"192.168.200.78","path":"/ssd-stor/primary","port":2049,"url":"NetworkFilesystem://192.168.200.78/ssd-stor/primary/?ROLE=Primary&STOREUUID=64d28c3f-4953-3f6c-b8f7-5858029226d7"}},"name":"ROOT-68","size":214748364800,"path":"6914e6e6-8529-4a19-b4cf-efd620efce39","volumeId":72,"vmName":"i-4-68-VM","accountId":4,"format":"VHD","provisioningType":"THIN","id":72,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"c827e745-4209-4615-802b-9759e94a588e","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"64d28c3f-4953-3f6c-b8f7-5858029226d7","id":3,"poolType":"NetworkFilesystem","host":"192.168.200.78","path":"/ssd-stor/primary","port":2049,"url":"NetworkFilesystem://192.168.200.78/ssd-stor/primary/?ROLE=Primary&STOREUUID=64d28c3f-4953-3f6c-b8f7-5858029226d7"}},"vmName":"i-4-68-VM","name":"test15nov","hypervisorType":"XenServer","id":171,"quiescevm":false,"parents":["c827e745-4209-4615-802b-9759e94a588e"],"physicalSize":0}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/4/72","volume":{"uuid":"f4a5f87e-232b-4956-af21-a3bc232cf47b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"64d28c3f-4953-3f6c-b8f7-5858029226d7","id":3,"poolType":"NetworkFilesystem","host":"192.168.200.78","path":"/ssd-stor/primary","port":2049,"url":"NetworkFilesystem://192.168.200.78/ssd-stor/primary/?ROLE=Primary&STOREUUID=64d28c3f-4953-3f6c-b8f7-5858029226d7"}},"name":"ROOT-68","size":214748364800,"path":"6914e6e6-8529-4a19-b4cf-efd620efce39","volumeId":72,"vmName":"i-4-68-VM","accountId":4,"format":"VHD","provisioningType":"THIN","id":72,"deviceId":0,"hypervisorType":"XenServer"},"parentSnapshotPath":"snapshots/4/72/eaa25c94-a322-4823-9a64-8b39dd6d2736","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://192.168.234.78/sata-stor/secondary","_role":"Image"}},"vmName":"i-4-68-VM","name":"test15nov","hypervisorType":"XenServer","id":171,"quiescevm":false,"parents":["snapshots/4/72/afca7b2b-2877-44bf-9a0a-6356965d0360","snapshots/4/72/60a2388b-ef18-487a-a87d-8c73ddbb1498","snapshots/4/72/85f183ba-b433-4c99-a0f7-c486c3c143a4","snapshots/4/72/fcb5445d-1a1e-4eae-a6c2-58dfa7969948","snapshots/4/72/e7bbceac-5e30-4201-a880-e170e0756264","snapshots/4/72/149d9cc8-821f-485c-8781-c4e411d112bb","snapshots/4/72/95977384-ccdc-4472-8e69-ad2610a6fd78","snapshots/4/72/eaa25c94-a322-4823-9a64-8b39dd6d2736"],"physicalSize":0}},"executeInSequence":true,"options":{"fullSnapshot":"false"},"options2":{},"wait":21600}}] } 2016-11-15 16:29:21,004 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-354:ctx-afb8860b) (logid:072c65e6) Seq 1-7805019629211344944: Executing request 2016-11-15 16:29:22,117 DEBUG [c.c.a.ApiServlet] (catalina-exec-21:ctx-5c2036f4) (logid:f40a5259) ===START=== 172.16.192.198 -- GET command=queryAsyncJobResult&jobId=bd1d8a8f-8820-4417-970c-0525c0ec332b&response=json&_=1479227445521 2016-11-15 16:29:22,132 DEBUG [c.c.a.ApiServlet] (catalina-exec-21:ctx-5c2036f4 ctx-120bc3d9) (logid:f40a5259) ===END=== 172.16.192.198 -- GET command=queryAsyncJobResult&jobId=bd1d8a8f-8820-4417-970c-0525c0ec332b&response=json&_=1479227445521 2016-11-15 16:29:22,233 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-1:null) (logid:) Ping from 10(v-42-VM) 2016-11-15 16:29:24,064 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-0ff63a56) (logid:1cd3a497) Begin cleanup expired async-jobs 2016-11-15 16:29:24,068 INFO [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-0ff63a56) (logid:1cd3a497) End cleanup expired async-jobs 2016-11-15 16:29:24,387 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-354:ctx-afb8860b) (logid:bd1d8a8f) Task failed! Task record: uuid: 159de760-e611-b0c9-07f7-9e7817fa5b62 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [] currentOperations: {} created: Tue Nov 15 16:30:45 GMT 2016 finished: Tue Nov 15 16:30:47 GMT 2016 status: failure residentOn: com.xensource.xenapi.Host@53eb80d4 progress: 1.0 type: result: errorInfo: [INTERNAL_ERROR, End_of_file] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] 2016-11-15 16:29:24,406 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-354:ctx-afb8860b) (logid:bd1d8a8f) unable to destroy task(OpaqueRef:74249017-228a-370d-de5d-0910ce4aaefe) due to You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given. 2016-11-15 16:29:24,406 DEBUG [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-354:ctx-afb8860b) (logid:bd1d8a8f) Exception in backupsnapshot stage due to Task failed! Task record: uuid: 159de760-e611-b0c9-07f7-9e7817fa5b62 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [] currentOperations: {} created: Tue Nov 15 16:30:45 GMT 2016 finished: Tue Nov 15 16:30:47 GMT 2016 status: failure residentOn: com.xensource.xenapi.Host@53eb80d4 progress: 1.0 type: result: errorInfo: [INTERNAL_ERROR, End_of_file] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] 2016-11-15 16:29:24,406 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-354:ctx-afb8860b) (logid:bd1d8a8f) Host 172.16.234.77 OpaqueRef:8fac1f71-bcd8-9ec6-fc9d-21c3d1046b3e: Removing SR 2016-11-15 16:29:24,433 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-354:ctx-afb8860b) (logid:bd1d8a8f) Host 172.16.234.77 OpaqueRef:b6b7bda0-cdff-eb6e-59f9-6eb21bbe851c: Unplugging pbd 2016-11-15 16:29:25,117 DEBUG [c.c.a.ApiServlet] (catalina-exec-2:ctx-3359158f) (logid:d5e27f65) ===START=== 172.16.192.198 -- GET command=queryAsyncJobResult&jobId=bd1d8a8f-8820-4417-970c-0525c0ec332b&response=json&_=1479227448521 2016-11-15 16:29:25,131 DEBUG [c.c.a.ApiServlet] (catalina-exec-2:ctx-3359158f ctx-860d5606) (logid:d5e27f65) ===END=== 172.16.192.198 -- GET command=queryAsyncJobResult&jobId=bd1d8a8f-8820-4417-970c-0525c0ec332b&response=json&_=1479227448521 2016-11-15 16:29:27,522 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-354:ctx-afb8860b) (logid:bd1d8a8f) Host 172.16.234.77 OpaqueRef:8fac1f71-bcd8-9ec6-fc9d-21c3d1046b3e: Forgetting 2016-11-15 16:29:27,524 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-354:ctx-afb8860b) (logid:bd1d8a8f) BackupSnapshot Failed due to Exception in backupsnapshot stage due to Task failed! Task record: uuid: 159de760-e611-b0c9-07f7-9e7817fa5b62 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [] currentOperations: {} created: Tue Nov 15 16:30:45 GMT 2016 finished: Tue Nov 15 16:30:47 GMT 2016 status: failure residentOn: com.xensource.xenapi.Host@53eb80d4 progress: 1.0 type: result: errorInfo: [INTERNAL_ERROR, End_of_file] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] com.cloud.utils.exception.CloudRuntimeException: Exception in backupsnapshot stage due to Task failed! Task record: uuid: 159de760-e611-b0c9-07f7-9e7817fa5b62 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [] currentOperations: {} created: Tue Nov 15 16:30:45 GMT 2016 finished: Tue Nov 15 16:30:47 GMT 2016 status: failure residentOn: com.xensource.xenapi.Host@53eb80d4 progress: 1.0 type: result: errorInfo: [INTERNAL_ERROR, End_of_file] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] at com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:370) at com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:528) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:99) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:54) at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:36) at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStorageSubSystemCommandWrapper.execute(CitrixStorageSubSystemCommandWrapper.java:30) at com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122) at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: Task failed! Task record: uuid: 159de760-e611-b0c9-07f7-9e7817fa5b62 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [] currentOperations: {} created: Tue Nov 15 16:30:45 GMT 2016 finished: Tue Nov 15 16:30:47 GMT 2016 status: failure residentOn: com.xensource.xenapi.Host@53eb80d4 progress: 1.0 type: result: errorInfo: [INTERNAL_ERROR, End_of_file] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] at com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:461) at com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:353) at com.cloud.hypervisor.xenserver.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:528) ... 19 more 2016-11-15 16:29:27,525 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-354:ctx-afb8860b) (logid:bd1d8a8f) Seq 1-7805019629211344944: Response Received: 2016-11-15 16:29:27,525 DEBUG [c.c.a.t.Request] (DirectAgent-354:ctx-afb8860b) (logid:bd1d8a8f) Seq 1-7805019629211344944: Processing: { Ans: , MgmtId: 14038011095064, via: 1(xenserver-2917), Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"BackupSnapshot Failed due to Exception in backupsnapshot stage due to Task failed! Task record: uuid: 159de760-e611-b0c9-07f7-9e7817fa5b62\n nameLabel: Async.VDI.copy\n nameDescription: \n allowedOperations: []\n currentOperations: {}\n created: Tue Nov 15 16:30:45 GMT 2016\n finished: Tue Nov 15 16:30:47 GMT 2016\n status: failure\n residentOn: com.xensource.xenapi.Host@53eb80d4\n progress: 1.0\n type: \n result: \n errorInfo: [INTERNAL_ERROR, End_of_file]\n otherConfig: {}\n subtaskOf: com.xensource.xenapi.Task@aaf13f6f\n subtasks: []\n","wait":0}}] } 2016-11-15 16:29:27,525 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-354:ctx-afb8860b) (logid:bd1d8a8f) Seq 1-7805019629211344944: No more commands found 2016-11-15 16:29:27,525 DEBUG [c.c.a.t.Request] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Seq 1-7805019629211344944: Received: { Ans: , MgmtId: 14038011095064, via: 1(xenserver-2917), Ver: v1, Flags: 110, { CopyCmdAnswer } } 2016-11-15 16:29:27,539 DEBUG [c.c.s.s.SnapshotManagerImpl] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Failed to create snapshot com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Exception in backupsnapshot stage due to Task failed! Task record: uuid: 159de760-e611-b0c9-07f7-9e7817fa5b62 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [] currentOperations: {} created: Tue Nov 15 16:30:45 GMT 2016 finished: Tue Nov 15 16:30:47 GMT 2016 status: failure residentOn: com.xensource.xenapi.Host@53eb80d4 progress: 1.0 type: result: errorInfo: [INTERNAL_ERROR, End_of_file] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:285) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:155) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:322) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1007) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy189.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy194.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-11-15 16:29:27,546 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Take snapshot: 72 failed com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1023) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy189.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy194.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Exception in backupsnapshot stage due to Task failed! Task record: uuid: 159de760-e611-b0c9-07f7-9e7817fa5b62 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [] currentOperations: {} created: Tue Nov 15 16:30:45 GMT 2016 finished: Tue Nov 15 16:30:47 GMT 2016 status: failure residentOn: com.xensource.xenapi.Host@53eb80d4 progress: 1.0 type: result: errorInfo: [INTERNAL_ERROR, End_of_file] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:285) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:155) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:322) ... 45 more 2016-11-15 16:29:27,547 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to take snapshot 2016-11-15 16:29:27,557 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376 ctx-1bec1fd4) (logid:bd1d8a8f) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to take snapshot 2016-11-15 16:29:27,557 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 68, job origin: 1375 2016-11-15 16:29:27,558 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Unable to complete AsyncJobVO {id:1376, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAARaAAlxdWllc2NlVm1MAAhwb2xpY3lJZHQAEExqYXZhL2xhbmcvTG9uZztMAApzbmFwc2hvdElkcQB-AAFMAAh2b2x1bWVJZHEAfgABeHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAQAAAAAAAAAAgAAAAAAAABEdAAUVm9sdW1lQXBpU2VydmljZUltcGwAc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAAAc3EAfgAGAAAAAAAAAKtzcQB-AAYAAAAAAAAASA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 14038011095064, completeMsid: null, lastUpdated: null, lastPolled: null, created: Tue Nov 15 16:29:19 GMT 2016}, job origin:1375 com.cloud.utils.exception.CloudRuntimeException: Failed to take snapshot at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1594) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2107) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2899) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2907) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy194.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1023) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ... 42 more Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Exception in backupsnapshot stage due to Task failed! Task record: uuid: 159de760-e611-b0c9-07f7-9e7817fa5b62 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [] currentOperations: {} created: Tue Nov 15 16:30:45 GMT 2016 finished: Tue Nov 15 16:30:47 GMT 2016 status: failure residentOn: com.xensource.xenapi.Host@53eb80d4 progress: 1.0 type: result: errorInfo: [INTERNAL_ERROR, End_of_file] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [] at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:285) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:155) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:322) ... 45 more 2016-11-15 16:29:27,583 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Complete async job-1376, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwc3EAfgAAc3EAfgAAcQB-AAp0ArRCYWNrdXBTbmFwc2hvdCBGYWlsZWQgZHVlIHRvIEV4Y2VwdGlvbiBpbiBiYWNrdXBzbmFwc2hvdCBzdGFnZSBkdWUgdG8gVGFzayBmYWlsZWQhIFRhc2sgcmVjb3JkOiAgICAgICAgICAgICAgICAgdXVpZDogMTU5ZGU3NjAtZTYxMS1iMGM5LTA3ZjctOWU3ODE3ZmE1YjYyCiAgICAgICAgICAgbmFtZUxhYmVsOiBBc3luYy5WREkuY29weQogICAgIG5hbWVEZXNjcmlwdGlvbjogCiAgIGFsbG93ZWRPcGVyYXRpb25zOiBbXQogICBjdXJyZW50T3BlcmF0aW9uczoge30KICAgICAgICAgICAgIGNyZWF0ZWQ6IFR1ZSBOb3YgMTUgMTY6MzA6NDUgR01UIDIwMTYKICAgICAgICAgICAgZmluaXNoZWQ6IFR1ZSBOb3YgMTUgMTY6MzA6NDcgR01UIDIwMTYKICAgICAgICAgICAgICBzdGF0dXM6IGZhaWx1cmUKICAgICAgICAgIHJlc2lkZW50T246IGNvbS54ZW5zb3VyY2UueGVuYXBpLkhvc3RANTNlYjgwZDQKICAgICAgICAgICAgcHJvZ3Jlc3M6IDEuMAogICAgICAgICAgICAgICAgdHlwZTogPG5vbmUvPgogICAgICAgICAgICAgIHJlc3VsdDogCiAgICAgICAgICAgZXJyb3JJbmZvOiBbSU5URVJOQUxfRVJST1IsIEVuZF9vZl9maWxlXQogICAgICAgICBvdGhlckNvbmZpZzoge30KICAgICAgICAgICBzdWJ0YXNrT2Y6IGNvbS54ZW5zb3VyY2UueGVuYXBpLlRhc2tAYWFmMTNmNmYKICAgICAgICAgICAgc3VidGFza3M6IFtdCnVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAMHNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAFTAAIZmlsZU5hbWVxAH4ABUwACm1ldGhvZE5hbWVxAH4ABXhwAAABHXQAOm9yZy5hcGFjaGUuY2xvdWRzdGFjay5zdG9yYWdlLnNuYXBzaG90LlNuYXBzaG90U2VydmljZUltcGx0ABhTbmFwc2hvdFNlcnZpY2VJbXBsLmphdmF0AA5iYWNrdXBTbmFwc2hvdHNxAH4ADgAAAJt0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2suc3RvcmFnZS5zbmFwc2hvdC5YZW5zZXJ2ZXJTbmFwc2hvdFN0cmF0ZWd5dAAeWGVuc2VydmVyU25hcHNob3RTdHJhdGVneS5qYXZhcQB-ABJzcQB-AA4AAAFCcQB-ABRxAH4AFXQADHRha2VTbmFwc2hvdHNxAH4ADgAAA-90AC5jb20uY2xvdWQuc3RvcmFnZS5zbmFwc2hvdC5TbmFwc2hvdE1hbmFnZXJJbXBsdAAYU25hcHNob3RNYW5hZ2VySW1wbC5qYXZhcQB-ABdzcQB-AA7____-dAAkc3VuLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsdAAdTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmF0AAdpbnZva2Uwc3EAfgAOAAAAPnEAfgAccQB-AB10AAZpbnZva2VzcQB-AA4AAAArdAAoc3VuLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAgc3EAfgAOAAAB8XQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ACBzcQB-AA4AAAE9dAAob3JnLnNwcmluZ2ZyYW1ld29yay5hb3Auc3VwcG9ydC5Bb3BVdGlsc3QADUFvcFV0aWxzLmphdmF0AB5pbnZva2VKb2lucG9pbnRVc2luZ1JlZmxlY3Rpb25zcQB-AA4AAAC3dAA8b3JnLnNwcmluZ2ZyYW1ld29yay5hb3AuZnJhbWV3b3JrLlJlZmxlY3RpdmVNZXRob2RJbnZvY2F0aW9udAAfUmVmbGVjdGl2ZU1ldGhvZEludm9jYXRpb24uamF2YXQAD2ludm9rZUpvaW5wb2ludHNxAH4ADgAAAJZxAH4ALHEAfgAtdAAHcHJvY2VlZHNxAH4ADgAAAFt0AD9vcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5pbnRlcmNlcHRvci5FeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3J0ACBFeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3IuamF2YXEAfgAgc3EAfgAOAAAArHEAfgAscQB-AC1xAH4AMHNxAH4ADgAAAMx0ADRvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5mcmFtZXdvcmsuSmRrRHluYW1pY0FvcFByb3h5dAAXSmRrRHluYW1pY0FvcFByb3h5LmphdmFxAH4AIHNxAH4ADv____90ABdjb20uc3VuLnByb3h5LiRQcm94eTE4OXBxAH4AF3NxAH4ADgAABjd0ADZvcmcuYXBhY2hlLmNsb3Vkc3RhY2suc3RvcmFnZS52b2x1bWUuVm9sdW1lU2VydmljZUltcGx0ABZWb2x1bWVTZXJ2aWNlSW1wbC5qYXZhcQB-ABdzcQB-AA4AAAg7dAAmY29tLmNsb3VkLnN0b3JhZ2UuVm9sdW1lQXBpU2VydmljZUltcGx0ABlWb2x1bWVBcGlTZXJ2aWNlSW1wbC5qYXZhdAAdb3JjaGVzdHJhdGVUYWtlVm9sdW1lU25hcHNob3RzcQB-AA4AAAtTcQB-AD5xAH4AP3EAfgBAc3EAfgAO_____nEAfgAccQB-AB1xAH4AHnNxAH4ADgAAAD5xAH4AHHEAfgAdcQB-ACBzcQB-AA4AAAArcQB-ACJxAH4AI3EAfgAgc3EAfgAOAAAB8XEAfgAlcQB-ACZxAH4AIHNxAH4ADgAAAGt0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AA4AAAtbcQB-AD5xAH4AP3EAfgBJc3EAfgAO_____nEAfgAccQB-AB1xAH4AHnNxAH4ADgAAAD5xAH4AHHEAfgAdcQB-ACBzcQB-AA4AAAArcQB-ACJxAH4AI3EAfgAgc3EAfgAOAAAB8XEAfgAlcQB-ACZxAH4AIHNxAH4ADgAAAT1xAH4AKHEAfgApcQB-ACpzcQB-AA4AAAC3cQB-ACxxAH4ALXEAfgAuc3EAfgAOAAAAlnEAfgAscQB-AC1xAH4AMHNxAH4ADgAAAFtxAH4AMnEAfgAzcQB-ACBzcQB-AA4AAACscQB-ACxxAH4ALXEAfgAwc3EAfgAOAAAAzHEAfgA2cQB-ADdxAH4AIHNxAH4ADv____90ABdjb20uc3VuLnByb3h5LiRQcm94eTE5NHBxAH4ASXNxAH4ADgAAAGZ0ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnNxAH4ADgAAAip0AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRzcQB-AA4AAAAxdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5zcQB-AA4AAAA4dABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxsc3EAfgAOAAAAZ3QAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AZXQAD2NhbGxXaXRoQ29udGV4dHNxAH4ADgAAADVxAH4AaHEAfgBldAAOcnVuV2l0aENvbnRleHRzcQB-AA4AAAAudAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-AGFxAH4AYnNxAH4ADgAAAfZxAH4AXHEAfgBdcQB-AGJzcQB-AA4AAAH_dAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-AGZzcQB-AA4AAAEKdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgBic3EAfgAOAAAEdnQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2Vyc3EAfgAOAAACaXQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4Ad3EAfgBic3EAfgAOAAAC6XQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgBic3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVMaXN0_A8lMbXsjhACAAFMAARsaXN0cQB-AAd4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbGxlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhxAH4Ag3gAABCadwgAAAAAAAAAAHh0ABlGYWlsZWQgdG8gY3JlYXRlIHNuYXBzaG90dXEAfgAMAAAALXNxAH4ADgAAA_9xAH4AGXEAfgAacQB-ABdzcQB-AA7____-cQB-ABxxAH4AHXEAfgAec3EAfgAOAAAAPnEAfgAccQB-AB1xAH4AIHNxAH4ADgAAACtxAH4AInEAfgAjcQB-ACBzcQB-AA4AAAHxcQB-ACVxAH4AJnEAfgAgc3EAfgAOAAABPXEAfgAocQB-AClxAH4AKnNxAH4ADgAAALdxAH4ALHEAfgAtcQB-AC5zcQB-AA4AAACWcQB-ACxxAH4ALXEAfgAwc3EAfgAOAAAAW3EAfgAycQB-ADNxAH4AIHNxAH4ADgAAAKxxAH4ALHEAfgAtcQB-ADBzcQB-AA4AAADMcQB-ADZxAH4AN3EAfgAgc3EAfgAO_____3EAfgA5cHEAfgAXc3EAfgAOAAAGN3EAfgA7cQB-ADxxAH4AF3NxAH4ADgAACDtxAH4APnEAfgA_cQB-AEBzcQB-AA4AAAtTcQB-AD5xAH4AP3EAfgBAc3EAfgAO_____nEAfgAccQB-AB1xAH4AHnNxAH4ADgAAAD5xAH4AHHEAfgAdcQB-ACBzcQB-AA4AAAArcQB-ACJxAH4AI3EAfgAgc3EAfgAOAAAB8XEAfgAlcQB-ACZxAH4AIHNxAH4ADgAAAGtxAH4AR3EAfgBIcQB-AElzcQB-AA4AAAtbcQB-AD5xAH4AP3EAfgBJc3EAfgAO_____nEAfgAccQB-AB1xAH4AHnNxAH4ADgAAAD5xAH4AHHEAfgAdcQB-ACBzcQB-AA4AAAArcQB-ACJxAH4AI3EAfgAgc3EAfgAOAAAB8XEAfgAlcQB-ACZxAH4AIHNxAH4ADgAAAT1xAH4AKHEAfgApcQB-ACpzcQB-AA4AAAC3cQB-ACxxAH4ALXEAfgAuc3EAfgAOAAAAlnEAfgAscQB-AC1xAH4AMHNxAH4ADgAAAFtxAH4AMnEAfgAzcQB-ACBzcQB-AA4AAACscQB-ACxxAH4ALXEAfgAwc3EAfgAOAAAAzHEAfgA2cQB-ADdxAH4AIHNxAH4ADv____9xAH4AVnBxAH4ASXNxAH4ADgAAAGZxAH4AWHEAfgBZcQB-AFpzcQB-AA4AAAIqcQB-AFxxAH4AXXEAfgBec3EAfgAOAAAAMXEAfgBgcQB-AGFxAH4AYnNxAH4ADgAAADhxAH4AZHEAfgBlcQB-AGZzcQB-AA4AAABncQB-AGhxAH4AZXEAfgBpc3EAfgAOAAAANXEAfgBocQB-AGVxAH4Aa3NxAH4ADgAAAC5xAH4AbXEAfgBhcQB-AGJzcQB-AA4AAAH2cQB-AFxxAH4AXXEAfgBic3EAfgAOAAAB_3EAfgBwcQB-AHFxAH4AZnNxAH4ADgAAAQpxAH4Ac3EAfgB0cQB-AGJzcQB-AA4AAAR2cQB-AHZxAH4Ad3EAfgB4c3EAfgAOAAACaXEAfgB6cQB-AHdxAH4AYnNxAH4ADgAAAulxAH4AfHEAfgB9cQB-AGJxAH4AgXgAABCadwgAAAAAAAAAAHh0ABdGYWlsZWQgdG8gdGFrZSBzbmFwc2hvdHVxAH4ADAAAACFzcQB-AA4AAAY6cQB-ADtxAH4APHEAfgAXc3EAfgAOAAAIO3EAfgA-cQB-AD9xAH4AQHNxAH4ADgAAC1NxAH4APnEAfgA_cQB-AEBzcQB-AA7____-cQB-ABxxAH4AHXEAfgAec3EAfgAOAAAAPnEAfgAccQB-AB1xAH4AIHNxAH4ADgAAACtxAH4AInEAfgAjcQB-ACBzcQB-AA4AAAHxcQB-ACVxAH4AJnEAfgAgc3EAfgAOAAAAa3EAfgBHcQB-AEhxAH4ASXNxAH4ADgAAC1txAH4APnEAfgA_cQB-AElzcQB-AA7____-cQB-ABxxAH4AHXEAfgAec3EAfgAOAAAAPnEAfgAccQB-AB1xAH4AIHNxAH4ADgAAACtxAH4AInEAfgAjcQB-ACBzcQB-AA4AAAHxcQB-ACVxAH4AJnEAfgAgc3EAfgAOAAABPXEAfgAocQB-AClxAH4AKnNxAH4ADgAAALdxAH4ALHEAfgAtcQB-AC5zcQB-AA4AAACWcQB-ACxxAH4ALXEAfgAwc3EAfgAOAAAAW3EAfgAycQB-ADNxAH4AIHNxAH4ADgAAAKxxAH4ALHEAfgAtcQB-ADBzcQB-AA4AAADMcQB-ADZxAH4AN3EAfgAgc3EAfgAO_____3EAfgBWcHEAfgBJc3EAfgAOAAAAZnEAfgBYcQB-AFlxAH4AWnNxAH4ADgAAAipxAH4AXHEAfgBdcQB-AF5zcQB-AA4AAAAxcQB-AGBxAH4AYXEAfgBic3EAfgAOAAAAOHEAfgBkcQB-AGVxAH4AZnNxAH4ADgAAAGdxAH4AaHEAfgBlcQB-AGlzcQB-AA4AAAA1cQB-AGhxAH4AZXEAfgBrc3EAfgAOAAAALnEAfgBtcQB-AGFxAH4AYnNxAH4ADgAAAfZxAH4AXHEAfgBdcQB-AGJzcQB-AA4AAAH_cQB-AHBxAH4AcXEAfgBmc3EAfgAOAAABCnEAfgBzcQB-AHRxAH4AYnNxAH4ADgAABHZxAH4AdnEAfgB3cQB-AHhzcQB-AA4AAAJpcQB-AHpxAH4Ad3EAfgBic3EAfgAOAAAC6XEAfgB8cQB-AH1xAH4AYnEAfgCBeAAAEJp3CAAAAAAAAAAAeA 2016-11-15 16:29:27,584 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Publish async job-1376 complete on message bus 2016-11-15 16:29:27,584 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Wake up jobs related to job-1376 2016-11-15 16:29:27,584 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Update db status for job-1376 2016-11-15 16:29:27,586 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Wake up jobs joined with job-1376 and disjoin all subjobs created from job- 1376 2016-11-15 16:29:27,590 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-1376 2016-11-15 16:29:27,591 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-25:ctx-e29c7751 job-1375/job-1376) (logid:bd1d8a8f) Remove job-1376 from job monitoring 2016-11-15 16:29:27,598 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:ctx-8acb10b8 job-1375) (logid:bd1d8a8f) Complete async job-1375, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 72"} 2016-11-15 16:29:27,599 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:ctx-8acb10b8 job-1375) (logid:bd1d8a8f) Publish async job-1375 complete on message bus 2016-11-15 16:29:27,599 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:ctx-8acb10b8 job-1375) (logid:bd1d8a8f) Wake up jobs related to job-1375 2016-11-15 16:29:27,599 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:ctx-8acb10b8 job-1375) (logid:bd1d8a8f) Update db status for job-1375 2016-11-15 16:29:27,600 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:ctx-8acb10b8 job-1375) (logid:bd1d8a8f) Wake up jobs joined with job-1375 and disjoin all subjobs created from job- 1375 2016-11-15 16:29:27,601 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-24:ctx-8acb10b8 job-1375) (logid:bd1d8a8f) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-1375 2016-11-15 16:29:27,601 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-24:ctx-8acb10b8 job-1375) (logid:bd1d8a8f) Remove job-1375 from job monitoring 2016-11-15 16:29:28,115 DEBUG [c.c.a.ApiServlet] (catalina-exec-9:ctx-923cdeff) (logid:256dbcc4) ===START=== 172.16.192.198 -- GET command=queryAsyncJobResult&jobId=bd1d8a8f-8820-4417-970c-0525c0ec332b&response=json&_=1479227451520 2016-11-15 16:29:28,129 DEBUG [c.c.a.ApiServlet] (catalina-exec-9:ctx-923cdeff ctx-3c7a9f7e) (logid:256dbcc4) ===END=== 172.16.192.198 -- GET command=queryAsyncJobResult&jobId=bd1d8a8f-8820-4417-970c-0525c0ec332b&response=json&_=1479227451520 2016-11-15 16:29:28,380 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5a489134) (logid:b5159ad0) Resetting hosts suitable for reconnect 2016-11-15 16:29:28,381 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5a489134) (logid:b5159ad0) Completed resetting hosts suitable for reconnect 2016-11-15 16:29:28,381 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5a489134) (logid:b5159ad0) Acquiring hosts for clusters already owned by this management server 2016-11-15 16:29:28,382 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5a489134) (logid:b5159ad0) Completed acquiring hosts for clusters already owned by this management server 2016-11-15 16:29:28,382 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5a489134) (logid:b5159ad0) Acquiring hosts for clusters not owned by any management server 2016-11-15 16:29:28,383 DEBUG [c.c.h.d.HostDaoImpl] (ClusteredAgentManager Timer:ctx-5a489134) (logid:b5159ad0) Completed acquiring hosts for clusters not owned by any management server 2016-11-15 16:29:30,270 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-11:null) (logid:) SeqA 10-1096877: Processing Seq 10-1096877: { Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":42,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2016-11-15 16:29:30,272 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-11:null) (logid:) SeqA 10-1096877: Sending Seq 10-1096877: { Ans: , MgmtId: 14038011095064, via: 10, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }