I am, despite my best judgment and common sense, acting as a system administrator for a handful of projects/users, sometimes on my own, sometimes as part of a team. On the bright side, it's usually a great source of weird debugging/workaround stories, like this one.
With the new release of
proxmox
and with 30 minutes of free time, a sysadmin team I'm part of thought it would
be nice to upgrade a box together. An admin ran pve8to9 --full, changed the
repositories from bookworm to trixie, and ran apt dist-upgrade, and… their
screen was filled with WARNING: ld.so: invalid GLIBC_TUNABLES
`glibc.malloc.perturb=1,glibc.malloc.check=1,glibc.malloc.tcache_count=0':
ignored.. Panic. So they hit <C-c> to investigate what was happening.
Turns out, another admin threw those configuration options somewhere to
"harden" the glibc allocator, and they're now deprecated, hence the message. No
big deal, this can be fixed by removing those options, and running apt
dist-upgrade again. Or can it?
# apt dist-upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
libanyevent-perl : Depends: perlapi-5.40.0
Recommends: libasync-interrupt-perl but it is not installed
Recommends: libev-perl but it is not installed or
libevent-perl but it is not installed
Recommends: libguard-perl but it is not installed
libcommon-sense-perl : Depends: perl (< 5.36.1~) but 5.40.1-6 is installed
libnss-systemd : Depends: systemd (= 257.7-1) but 252.38-1~deb12u1 is installed
libpam-systemd : Depends: systemd (= 257.7-1) but 252.38-1~deb12u1 is installed
libsasl2-2 : Depends: libssl3t64 (>= 3.0.0) but it is not installed
Recommends: libsasl2-modules (>= 2.1.28+dfsg1-9) but it is not installed
libsasl2-modules-db : Depends: libdb5.3t64 but it is not installed
libsystemd-shared : Depends: libssl3t64 (>= 3.4.0) but it is not installed
openssl : Depends: libssl3t64 (>= 3.5.0) but it is not installed
perl : Depends: perl-base (= 5.40.1-6) but 5.36.0-7+deb12u2 is installed
Depends: libperl5.40 (= 5.40.1-6) but it is not installed
perl-modules-5.40 : Depends: perl-base (>= 5.40.1-1) but 5.36.0-7+deb12u2 is installed
systemd : Depends: libsystemd-shared (= 252.38-1~deb12u1) but 257.7-1 is installed
Depends: libsystemd0 (= 252.38-1~deb12u1) but 257.7-1 is installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
[100]
#
Alright, let's try apt --fix-broken install:
# apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... Done
The following packages were automatically installed and are no longer required:
[…]
75 upgraded, 64 newly installed, 50 to remove and 530 not upgraded.
17 not fully installed or removed.
Need to get 0 B/118 MB of archives.
After this operation, 215 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
W: (pve-apt-hook) !! WARNING !!
W: (pve-apt-hook) You are attempting to remove the meta-package 'proxmox-ve'!
W: (pve-apt-hook)
W: (pve-apt-hook) If you really want to permanently remove 'proxmox-ve' from your system, run the following command
W: (pve-apt-hook) touch '/please-remove-proxmox-ve'
W: (pve-apt-hook) run apt purge proxmox-ve to remove the meta-package
W: (pve-apt-hook) and repeat your apt invocation.
W: (pve-apt-hook)
W: (pve-apt-hook) If you are unsure why 'proxmox-ve' would be removed, please verify
W: (pve-apt-hook) - your APT repository settings
W: (pve-apt-hook) - that you are using 'apt full-upgrade' to upgrade your system
E: Sub-process /usr/share/proxmox-ve/pve-apt-hook returned an error code (1)
E: Failure running script /usr/share/proxmox-ve/pve-apt-hook
[100]
#
We're absolutely not going to do that, so let's use apt-mark to tell apt
to leave proxmox-ve alone, and try the whole manoeuvre again:
# apt-mark hold proxmox-ve
proxmox-ve set on hold.
# apt --fix-broken install
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Correcting dependencies... failed.
The following packages have unmet dependencies:
libanyevent-perl : Depends: perlapi-5.40.0
Recommends: libasync-interrupt-perl but it is not installed
Recommends: libev-perl but it is not installed or
libevent-perl but it is not installed
Recommends: libguard-perl but it is not installed
libcommon-sense-perl : Depends: perl (< 5.36.1~) but 5.40.1-6 is installed
libnss-systemd : Depends: systemd (= 257.7-1) but 252.38-1~deb12u1 is installed
libpam-systemd : Depends: systemd (= 257.7-1) but 252.38-1~deb12u1 is installed
libsasl2-2 : Depends: libssl3t64 (>= 3.0.0) but it is not installed
Recommends: libsasl2-modules (>= 2.1.28+dfsg1-9) but it is not installed
libsasl2-modules-db : Depends: libdb5.3t64 but it is not installed
libsystemd-shared : Depends: libssl3t64 (>= 3.4.0) but it is not installed
openssl : Depends: libssl3t64 (>= 3.5.0) but it is not installed
perl : Depends: perl-base (= 5.40.1-6) but 5.36.0-7+deb12u2 is installed
Depends: libperl5.40 (= 5.40.1-6) but it is not installed
perl-modules-5.40 : Depends: perl-base (>= 5.40.1-1) but 5.36.0-7+deb12u2 is installed
systemd : Depends: libsystemd-shared (= 252.38-1~deb12u1) but 257.7-1 is installed
Depends: libsystemd0 (= 252.38-1~deb12u1) but 257.7-1 is installed
E: Error, pkgProblemResolver::Resolve generated breaks, this may be caused by held packages.
E: Unable to correct dependencies
[100]
#
You might think that using apt-mark hold for all the packages with unmet
dependencies is the way to go, but you would be wrong, as it doesn't fix shit.
You can check apt's documentation and ask the collective hivemind of the
internet, but we didn't find anything to solve our problem. Then we realised
that apt has a graph of all packages and their dependencies stored somewhere
on the filesystem, and we found it via strace -f -e openat 2>&1 apt install
--fix-missing | grep -oP "\"(.*)\"" | sort | uniq -c: it's /var/lib/dpkg/status.
We can now edit it to remove the Depends: field of every package with broken
dependencies to appease apt's dependencies solver via this big fat lie. Once done:
# apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
[…]
Setting up ca-certificates (20250419) ...
Updating certificates in /etc/ssl/certs...
openssl: /lib/x86_64-linux-gnu/libssl.so.3: version `OPENSSL_3.4.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libssl.so.3: version `OPENSSL_3.2.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.3.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.5.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.4.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.2.0' not found (required by openssl)
dpkg: error processing package ca-certificates (--configure):
installed ca-certificates package post-installation script subprocess returned error exit status 1
Setting up busybox (1:1.37.0-6+b3) ...
Setting up libklibc:amd64 (2.0.14-1) ...
Setting up novnc-pve (1.6.0-3) ...
Setting up libz3-4:amd64 (4.13.3-1) ...
Setting up xz-utils (5.8.1-1) ...
Setting up libfribidi0:amd64 (1.0.16-1) ...
Setting up libopus0:amd64 (1.5.2-2) ...
Setting up libquadmath0:amd64 (14.2.0-19) ...
dpkg: dependency problems prevent configuration of python3-requests:
python3-requests depends on python3-certifi; however:
Package python3-certifi is not configured yet.
python3-requests depends on ca-certificates; however:
Package ca-certificates is not configured yet.
dpkg: error processing package python3-requests (--configure):
dependency problems - leaving unconfigured
Setting up libproc2-0:amd64 (2:4.0.4-9) ...
Setting up proxmox-backup-restore-image (1.0.0) ...
Setting up ssl-cert (1.1.3) ...
openssl: /lib/x86_64-linux-gnu/libssl.so.3: version `OPENSSL_3.4.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libssl.so.3: version `OPENSSL_3.2.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.3.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.5.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.4.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.2.0' not found (required by openssl)
Signature algorithm of /etc/ssl/certs/ssl-cert-snakeoil.pem is not sha256. Recreating.
Could not create certificate. Openssl output was:
openssl: /lib/x86_64-linux-gnu/libssl.so.3: version `OPENSSL_3.4.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libssl.so.3: version `OPENSSL_3.2.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.3.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.5.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.4.0' not found (required by openssl)
openssl: /lib/x86_64-linux-gnu/libcrypto.so.3: version `OPENSSL_3.2.0' not found (required by openssl)
dpkg: error processing package ssl-cert (--configure):
installed ssl-cert package post-installation script subprocess returned error exit status 1
Setting up libxcb-sync1:amd64 (1.17.0-2+b1) ...
Setting up dtach (0.9-7) ...
Setting up lsb-release (12.1-1) ...
Setting up dbus-system-bus-common (1.16.2-2) ...
systemd-sysusers: error while loading shared libraries: libsystemd-shared-252.so: cannot open shared object file: No such file or directory
dpkg: error processing package dbus-system-bus-common (--configure):
installed dbus-system-bus-common package post-installation script subprocess returned error exit status 127
Setting up libconfuse2:amd64 (3.3-4) ...
Setting up libisl23:amd64 (0.27-1) ...
Setting up libcpg4:amd64 (3.1.9-pve2) ...
Setting up libcmap4:amd64 (3.1.9-pve2) ...
Setting up libpulse0:amd64 (17.0+dfsg1-2+b1) ...
dpkg: dependency problems prevent configuration of python3-pip-whl:
python3-pip-whl depends on ca-certificates; however:
Package ca-certificates is not configured yet.
dpkg: error processing package python3-pip-whl (--configure):
dependency problems - leaving unconfigured
Setting up libquorum5:amd64 (3.1.9-pve2) ...
Setting up iptables (1.8.11-2) ...
Setting up fontconfig-config (2.15.0-2.3) ...
Setting up uidmap (1:4.17.4-2) ...
dpkg: dependency problems prevent configuration of cron:
cron depends on cron-daemon-common; however:
Package cron-daemon-common is not configured yet.
dpkg: error processing package cron (--configure):
dependency problems - leaving unconfigured
Setting up python3-typeguard (4.4.2-1) ...
Setting up rsyslog (8.2504.0-1) ...
[…]
dpkg: error processing package pve-manager (--configure):
dependency problems - leaving triggers unprocessed
Processing triggers for man-db (2.11.2-2) ...
Processing triggers for sgml-base (1.31+nmu1) ...
Errors were encountered while processing:
ca-certificates
python3-certifi
ssl-cert
python3-requests
dbus-system-bus-common
cron-daemon-common
liblwp-protocol-https-perl
libwww-perl
python3-pip-whl
cron
libpve-common-perl
libproxmox-acme-perl
python3-pyvmomi
libpve-apiclient-perl
dbus
libpve-guest-common-perl
libpve-storage-perl
libpve-access-control
libpve-cluster-perl
libpve-http-server-perl
pve-ha-manager
libpve-cluster-api-perl
libpve-notify-perl
pve-manager
E: Sub-process /usr/bin/dpkg returned an error code (1)
[100]
#
Things are still broken, but less so, at least apt is able to
upgrade/install/manage/… packages. Now it's simply a matter of running dpkg
--force-all --configure -a, apt-get install --fix-missing and apt install
<PACKAGE> (with <PACKAGE> being things listed in the Errors were
encountered while processing: errors from the other commands) until you get no
more errors. At this point, you can reopen /var/lib/dpkg/status, uncomment
the lines you commented if they're still present, and resume your apt
dist-upgrade. Still, it's sad that in 2025 a package manager is a simple
SIGINT away from being completely broken.