VirtualBox, ulimit, open files and depmod.

Recently I decided that systemd wasn't for me. It currently breaks Cron on Debian testing (bad) and logs in Binary (stupid) and only gains me 2 seconds on startup! So I removed it and placed SystemV back. One of the parts was temporarily removing Dbus. This removed all sub-parts including my VirtualBox install. Knew what I had and how to reconstruct, so purge and then install.

On reinstall vitrualbox-dkms broke. In particular, looking at the log, it wasn't compiling the kernel modules for VirtualBox

make: Entering directory `/usr/src/linux-headers-3.13.0-24-generic'
LD /var/lib/dkms/vboxhost/4.2.16/build/built-in.o
...
CC [M] /var/lib/dkms/vboxhost/4.2.16/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.o
/var/lib/dkms/vboxhost/4.2.16/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c: In function ‘rtR0MemObjNativeMapUser’:
/var/lib/dkms/vboxhost/4.2.16/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.c:1539:26: error: ‘struct mm_struct’ has no member named ‘numa_next_reset’
pTask->mm->numa_next_reset = jiffies + 0x7fffffffffffffffUL;
^
make[2]: *** [/var/lib/dkms/vboxhost/4.2.16/build/vboxdrv/r0drv/linux/memobj-r0drv-linux.o] Error 1
make[1]: *** [/var/lib/dkms/vboxhost/4.2.16/build/vboxdrv] Error 2
make: *** [_module_/var/lib/dkms/vboxhost/4.2.16/build] Error 2
make: Leaving directory `/usr/src/linux-headers-3.13.0-24-generic'

Ok, update to latest versions of all software and retry.

Setting up virtualbox-dkms (4.3.14-dfsg-1) ...
Removing old virtualbox-4.3.14 DKMS files...

-------- Uninstall Beginning --------
Module:  virtualbox
Version: 4.3.14
Kernel:  3.13-1-amd64 (x86_64)
-------------------------------------

...

vboxpci.ko:
 - Uninstallation
   - Deleting from: /lib/modules/3.13-1-amd64/updates/
 - Original module
   - No original module was found for this module on this kernel.
   - Use the dkms install command to reinstall any previous module version.

depmod....(bad exit status: 1)

DKMS: uninstall completed.
Error! Problems with depmod detected.  Automatically uninstalling this module.
DKMS: Install Failed (depmod problems).  Module rolled back to built state.
dpkg: error processing package virtualbox-dkms (--configure):
 subprocess installed post-installation script returned error exit status 6
Errors were encountered while processing:
 virtualbox-dkms

Tried a few things; tried manually removing the kernel modules, they didn't exist, but were being uninstalled.

$ find /lib/ -name vboxpci.ko
$

This was weird because I'ld already tried reinstalling a few times and the script seemed to be finding the modules. According to the log they were available and being stopped.

Looked at the available files.

$ cat /proc/sys/fs/file-nr
5600    0   248540

Weird, no free files. Tried upping this; (duckducked)

$ echo 780335 > /proc/sys/fs/file-max
$ sysctl -p
$ aptitude install virtualbox
...
Errors were encountered while processing:
 vitrualbox-dkms
$ /sbin/sysctl fs.file-nr
fs.file-nr = 5632   0   780335

Looked at lsof. Nothing stuck. Looked at inodes, ~4%. Closed all open programs except terminal, still had 0 free. Looked online and found that this was the new status-quo for the kernel. daf*ck! Tried greping vbox to see if there were any modules loaded.

$ depmod -a | grep vbox
... <many lines>
ERROR: failed to load symbols from /lib/modules/3.13-1-amd64/kernel/lib/oid_registry.ko: Too many open files
...
ERROR: failed to load symbols from /lib/modules/3.13-1-amd64/updates/vboxdrv.ko: Too many open files
FATAL: could not open directory /lib/modules/3.13-1-amd64: Too many open files

Some lines didn't have vbox! The pipe was breaking so badly that error handling wasn't working!

Problem changed.

The system is REALLY running out of files. Duckducked again, and a vmware site suggested increasing the ulimit. Checked my ulimit

$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 61338
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 61338
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Vmware site suggested 65536. So I man ulimit

ULIMIT(3)       Linux Programmer's Manual

...
DESCRIPTION
   Warning: This routine is obsolete.  Use getrlimit(2), setrlimit(2), and sysconf(3) instead.  For the shell command ulimit(), see bash(1).

ZZ

$ getrlimit
bash: getrlimit: command not found

Yep, ulimit was obsolete but apparently the only controller on my system. Went 8x my current size, some sites also suggested 4096.

$ ulimit -n 8192
$ sysctl -p

$ aptitude install virtualbox-dkms=4.3.14-dfsg-1 
The following partially installed packages will be configured:
  virtualbox-dkms 
No packages will be installed, upgraded, or removed.
0 packages upgraded, 0 newly installed, 0 to remove and 31 not upgraded.
Need to get 0 B of archives. After unpacking 0 B will be used.
Setting up virtualbox-dkms (4.3.14-dfsg-1) ...
Removing old virtualbox-4.3.14 DKMS files...

-------- Uninstall Beginning --------
Module:  virtualbox
Version: 4.3.14
Kernel:  3.13-1-amd64 (x86_64)
-------------------------------------

Status: This module version was INACTIVE for this kernel.
depmod....

DKMS: uninstall completed.

------------------------------
Deleting module version: 4.3.14
completely from the DKMS tree.
------------------------------
Done.
Loading new virtualbox-4.3.14 DKMS files...
First Installation: checking all kernels...
Building only for 3.13-1-amd64
Building initial module for 3.13-1-amd64
Done.

vboxdrv:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.13-1-amd64/updates/

vboxnetadp.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.13-1-amd64/updates/

vboxnetflt.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.13-1-amd64/updates/

vboxpci.ko:
Running module version sanity check.
 - Original module
   - No original module exists within this kernel
 - Installation
   - Installing to /lib/modules/3.13-1-amd64/updates/

sed: -e expression #1, char 6: unknown command: `m'
depmod....

DKMS: install completed.
[ ok ] Stopping VirtualBox kernel modules.
[ ok ] Starting VirtualBox kernel modules.

Installed.

Side note, Firefox was sometimes missing connections and timing out, ulimit -n 8192 has improved this.

Michael 20140829