現象:
虛拟機使用了一段時間,虛拟機關機,然後再開機出現錯誤;
背景日志:
Thread-570342::ERROR::2018-02-05 15:31:17,944::vm::761::virt.vm::(_startUnderlyingVm) vmId=`18988d57-d0d3-4ff5-9e3a-146f2de4b6f2`::The vm start process failed
Traceback (most recent call last):
File "/usr/share/vdsm/virt/vm.py", line 705, in _startUnderlyingVm
self._run()
File "/usr/share/vdsm/virt/vm.py", line 1944, in _run
self._connection.createXML(domxml, flags),
File "/usr/lib/python2.7/site-packages/vdsm/libvirtconnection.py", line 124, in wrapper
ret = f(*args, **kwargs)
File "/usr/lib/python2.7/site-packages/vdsm/utils.py", line 1313, in wrapper
return func(inst, *args, **kwargs)
File "/usr/lib64/python2.7/site-packages/libvirt.py", line 3611, in createXML
if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirtError: 内部錯誤:Process exited prior to exec: libvirt: Lock Driver 錯誤 : 為 sanlock 守護程序打開插槽失敗: 拒絕連接配接
Thread-570342::INFO::2018-02-05 15:31:17,947::vm::1332::virt.vm::(setDownStatus) vmId=`18988d57-d0d3-4ff5-9e3a-146f2de4b6f2`::Changed state to Down: 内部錯誤:Process exited prior to exec: libvirt: Lock Driver >錯誤 : 為 sanlock 守護程序打開插槽失敗: 拒絕連接配接 (code=1)
sanlock.log
2018-02-05 15:37:59+0800 6554608 [135834]: sanlock daemon started 3.2.4 host 3c4a8445-6b22-42df-9b06-91d9774dae1d.host01
2018-02-05 15:37:59+0800 6554608 [135834]: set scheduler RR|RESET_ON_FORK priority 99 failed: Operation not permitted
解決方案:
重新啟動libvirtd服務,因為libvirt裡面使用了sanlock。
問題排查:
sanlock裡面的錯誤,sched_setscheduler(2)引起了sanlock沒有實時排程,當機器出現高負載IO的時候時得sanlock丢失了它的deadlines