Recovering from rm /*

...and received a terrifying output like this:

root@victim:/# rm /*
rm: cannot remove '/boot': Is a directory
rm: cannot remove '/dev': Is a directory
rm: cannot remove '/etc': Is a directory
rm: cannot remove '/home': Is a directory
rm: cannot remove '/media': Is a directory
rm: cannot remove '/mnt': Is a directory
rm: cannot remove '/opt': Is a directory
rm: cannot remove '/proc': Is a directory
rm: cannot remove '/root': Is a directory
rm: cannot remove '/run': Is a directory
rm: cannot remove '/srv': Is a directory
rm: cannot remove '/sys': Is a directory
rm: cannot remove '/tmp': Is a directory
rm: cannot remove '/usr': Is a directory
rm: cannot remove '/var': Is a directory

"Good thing it wasn't rm -r" – you might have thought. Clearly, it is. But that's all the good news, since your system is now pretty much unusable. You cannot run even basic shell utilities anymore.

root@victim:/# ls
bash: /usr/bin/ls: cannot execute: required file not found
root@victim:/# whoami
bash: /usr/bin/whoami: cannot execute: required file not found
root@victim:/# bash
bash: /usr/bin/bash: cannot execute: required file not found

What happened?

Although you didn't type rm -rf and avoided erasing your entire filesystem, you did remove the vital symlinks that sit in your root directory.

The system used in the example is Debian 12 on x86-64. It has four symlinks in the root directory.

root@victim:/# ls -la / | grep ^l
lrwxrwxrwx   1 root   root       8 Apr 10 11:03 bin -> /usr/bin
lrwxrwxrwx   1 root   root       8 Apr 10 11:03 lib -> /usr/lib
lrwxrwxrwx   1 root   root      10 Apr 10 11:03 lib64 -> /usr/lib64
lrwxrwxrwx   1 root   root       9 Apr 10 11:03 sbin -> /usr/sbin

/lib and /lib64 are the worst to lose, since your system expects to have a dynamic loader at /lib{,64}/ld-linux-*. On the example system, its full path is /lib64/ld-linux-x86-64.so.2.

The dynamic loader is the component of your system that is required to execute dynamically linked executables. To be specific, it's executables themselves who expect dynamic loader to exist in /lib or /lib64.

You could use ln -s /usr/lib64 /lib64 to restore the necessary symlink. However, the ln utility is also dynamically linked, so it won't execute either.

What to do now?

1. Find the real path of your dynamic loader. It should be located under or /usr/lib64/ or /usr/lib . You can't use ls utility for that, but you should be able to list directories with tab-completion.
2. Let's say you've found your linker at /usr/lib64/ld-linux-x86-64.so.2. Try running it. You should get:

root@victim:/# /usr/lib64/ld-linux-x86-64.so.2
/usr/lib64/ld-linux-x86-64.so.2: missing program name
Try '/usr/lib64/ld-linux-x86-64.so.2 --help' for more information.

By calling the dynamic loader directly, you can bypass one specified in your dynamic executables. The only thing is – you should specify the full path to your executable. Otherwise you will get an error:

root@victim:/# /usr/lib64/ld-linux-x86-64.so.2 echo test
echo: error while loading shared libraries: echo: cannot open shared object file

3. Restore the symlinks by running /usr/bin/ln with your dynamic loader.

/usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ln -s /lib /usr/lib
/usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ln -s /lib64 /usr/lib64
/usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ln -s /bin /usr/bin
/usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ln -s /sbin /usr/sbin

You should now have your system operational again.

4. Reboot for a safe measure. Your running daemons, like sshd or web servers could be in an unrecoverable state. They were likely not built to sustain such eventualities.

Oops, I've already closed the shell.

You should find another way to recreate the symlinks that you've deleted.

• If you have physical access to your computer, boot with Live CD. The easiest option is to create a bootable flash drive with ubuntu. Mount your hard drive and restore the symlinks from there.
• Some VDS providers have recovery procedures, such as booting in a rescue mode or using an ISO.
• Nothing worked? Well... I hope you've got some recent backups.

Can I fix it from a non-root shell?

Short answer: No.

If you tried running sudo using the same method, you might have already seen this error:

root@victim:/# /usr/lib64/ld-linux-x86-64.so.2 /usr/bin/sudo echo test
sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?

The nosuid guess here is misleading. In reality, the problem is caused by the mechanism that sudo uses internally to elevate privileges.

root@victim:/ ls -l /usr/bin/sudo
-rwsr-xr-x 1 root root 281624 Jun 27  2023 /usr/bin/sudo

/usr/bin/sudo is a setuid binary. You can verify it with ls command. This bit tells kernel to run this executable as a different user, specifically as one who owns the executable. In this case, it's a root user. Unfortunately, this contraption doesn't play well with our workaround.

The problem is that you are not running sudo using your kernel anymore. You are now using it to manually run your dynamic loader to load the sudo executable from there.

Dynamic loader cannot change the effective user by itself, since it's just an ordinary user space executable. The only instance that is capable of doing such substitutes is the kernel itself, which would only do so when loading setuid binaries. And your dynamic loader isn't one of these.

So, when sudo executable gets loaded and executed, it realizes that its effective user isn't root and complains. Unfortunately, as for my knowledge, there is no way of using setuid binaries with broken /lib64 symlink.

Conclusion

If you just so happened to type rm /* and you want to bring your system back to life without resorting to surgical intervention, then:

• Don't close your root shell;
• Locate your dynamic loader in /usr/lib64/ using tab-completion to list directories;
• Use it to restore the necessary symlinks:

/usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ln -s /lib /usr/lib
/usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ln -s /lib64 /usr/lib64
/usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ln -s /bin /usr/bin
/usr/lib64/ld-linux-x86-64.so.2 /usr/bin/ln -s /sbin /usr/sbin

• And never ever type rm in sudo su again. You know, one day this little password prompt will save your production server.