VirtualBox, ulimit, open files and depmod.
Recently I decided that systemd wasn't for me. It currently breaks Cron on Debian testing (bad) and logs in Binary (stupid) and only gains me 2 seconds on startup! So I removed it and placed SystemV back. One of the parts was temporarily removing Dbus. This removed all sub-parts including my VirtualBox install. Knew what I had and how to reconstruct, so purge and then install.
On reinstall vitrualbox-dkms broke. In particular, looking at the log, it wasn't compiling the kernel modules for VirtualBox
make: Entering directory `/usr/src/linux-headers-3.13.0-24-generic' LD /var/lib/dkms/vboxhost/4.2.16/build/built-in.o ... CC [M] /var/lib/dkms/vboxhost/4.2.16/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.o /var/lib/dkms/vboxhost/4.2.16/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c: In function ‘rtR0MemObjNativeMapUser’: /var/lib/dkms/vboxhost/4.2.16/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:1539:26: error: ‘struct mm_struct’ has no member named ‘numa_next_reset’ pTask->mm->numa_next_reset = jiffies + 0x7fffffffffffffffUL; ^ make: *** [/var/lib/dkms/vboxhost/4.2.16/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.o] Error 1 make: *** [/var/lib/dkms/vboxhost/4.2.16/build/vboxdrv] Error 2 make: *** [_module_/var/lib/dkms/vboxhost/4.2.16/build] Error 2 make: Leaving directory `/usr/src/linux-headers-3.13.0-24-generic'
Ok, update to latest versions of all software and retry.
Setting up virtualbox-dkms (4.3.14-dfsg-1) ... Removing old virtualbox-4.3.14 DKMS files... -------- Uninstall Beginning -------- Module: virtualbox Version: 4.3.14 Kernel: 3.13-1-amd64 (x86_64) ------------------------------------- ... vboxpci.ko: - Uninstallation - Deleting from: /lib/modules/3.13-1-amd64/updates/ - Original module - No original module was found for this module on this kernel. - Use the dkms install command to reinstall any previous module version. depmod....(bad exit status: 1) DKMS: uninstall completed. Error! Problems with depmod detected. Automatically uninstalling this module. DKMS: Install Failed (depmod problems). Module rolled back to built state. dpkg: error processing package virtualbox-dkms (--configure): subprocess installed post-installation script returned error exit status 6 Errors were encountered while processing: virtualbox-dkms
Tried a few things; tried manually removing the kernel modules, they didn't exist, but were being uninstalled.
$ find /lib/ -name vboxpci.ko $
This was weird because I'ld already tried reinstalling a few times and the script seemed to be finding the modules. According to the log they were available and being stopped.
Looked at the available files.
$ cat /proc/sys/fs/file-nr 5600 0 248540
Weird, no free files. Tried upping this; (duckducked)
$ echo 780335 > /proc/sys/fs/file-max $ sysctl -p $ aptitude install virtualbox ... Errors were encountered while processing: vitrualbox-dkms $ /sbin/sysctl fs.file-nr fs.file-nr = 5632 0 780335
Looked at lsof. Nothing stuck. Looked at inodes, ~4%. Closed all open programs except terminal, still had 0 free. Looked online and found that this was the new status-quo for the kernel. daf*ck! Tried greping vbox to see if there were any modules loaded.
$ depmod -a | grep vbox ... <many lines> ERROR: failed to load symbols from /lib/modules/3.13-1-amd64/kernel/lib/oid_registry.ko: Too many open files ... ERROR: failed to load symbols from /lib/modules/3.13-1-amd64/updates/vboxdrv.ko: Too many open files FATAL: could not open directory /lib/modules/3.13-1-amd64: Too many open files
Some lines didn't have vbox! The pipe was breaking so badly that error handling wasn't working!
The system is REALLY running out of files. Duckducked again, and a vmware site suggested increasing the ulimit. Checked my ulimit
$ ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 61338 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 61338 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited
Vmware site suggested 65536. So I man ulimit
ULIMIT(3) Linux Programmer's Manual ... DESCRIPTION Warning: This routine is obsolete. Use getrlimit(2), setrlimit(2), and sysconf(3) instead. For the shell command ulimit(), see bash(1). ZZ $ getrlimit bash: getrlimit: command not found
Yep, ulimit was obsolete but apparently the only controller on my system. Went 8x my current size, some sites also suggested 4096.
$ ulimit -n 8192 $ sysctl -p $ aptitude install virtualbox-dkms=4.3.14-dfsg-1 The following partially installed packages will be configured: virtualbox-dkms No packages will be installed, upgraded, or removed. 0 packages upgraded, 0 newly installed, 0 to remove and 31 not upgraded. Need to get 0 B of archives. After unpacking 0 B will be used. Setting up virtualbox-dkms (4.3.14-dfsg-1) ... Removing old virtualbox-4.3.14 DKMS files... -------- Uninstall Beginning -------- Module: virtualbox Version: 4.3.14 Kernel: 3.13-1-amd64 (x86_64) ------------------------------------- Status: This module version was INACTIVE for this kernel. depmod.... DKMS: uninstall completed. ------------------------------ Deleting module version: 4.3.14 completely from the DKMS tree. ------------------------------ Done. Loading new virtualbox-4.3.14 DKMS files... First Installation: checking all kernels... Building only for 3.13-1-amd64 Building initial module for 3.13-1-amd64 Done. vboxdrv: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/3.13-1-amd64/updates/ vboxnetadp.ko: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/3.13-1-amd64/updates/ vboxnetflt.ko: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/3.13-1-amd64/updates/ vboxpci.ko: Running module version sanity check. - Original module - No original module exists within this kernel - Installation - Installing to /lib/modules/3.13-1-amd64/updates/ sed: -e expression #1, char 6: unknown command: `m' depmod.... DKMS: install completed. [ ok ] Stopping VirtualBox kernel modules. [ ok ] Starting VirtualBox kernel modules.
Side note, Firefox was sometimes missing connections and timing out, ulimit -n 8192 has improved this.