RPM Database Corruption

Sean Horn -

If rpmdb claims it's corrupted, it could be just stale locks from a killed chef-client or other process. Make sure you figure out which PID was killed while it was trying to `rpm -qa` and then compare to the output of 

cd /var/lib/rpm
/usr/lib/rpm/rpmdb_stat -CA

Here is the lock portion of the output when there are no locks present. A lock identifier will look like this `12926/140090959366048` and appear in these two sections.

Locks grouped by lockers:
Locker Mode Count Status ----------------- Object ---------------
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Locks grouped by object:
Locker Mode Count Status ----------------- Object ---------------

If all processes that should be accessing the RPMDB are gone, but you still see locks in the rpmdb_stat output, then you probably have your "corruption" candidate.

Delete those locks with `rm -rf /var/lib/rpm/__db.0*`, then try your `rpm -qa` again. This time, reading should be successful unless there actually is a problem with the database's data.

You can reproduce this situation on CentOS 6.x with the following shell script

for i in {1..30}; do 
if [[ "$i" -eq 3 ]]; then 
   ( /bin/rpm -qa > /dev/null ; kill -9 $! ) & 
else 
  /bin/rpm -qa > /dev/null &  
fi  
done

That kill on the third rpm -qa doesn't allow the process to cleanly close the database resulting in instant corruption.

Any other attempts at a similar "read-only" command afterward result in output like

[root@chef-432 rpm]# rpm -qa
rpmdb: Thread/process 12926/140090959366048 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages index using db3 -  (-30974)
error: cannot open Packages database in /var/lib/rpm
rpmdb: Thread/process 12926/140090959366048 failed: Thread died in Berkeley DB library
error: db3 error(-30974) from dbenv->failchk: DB_RUNRECOVERY: Fatal error, run database recovery
error: cannot open Packages database in /var/lib/rpm

We have noticed the most issue with this when the yum_package resource is used in the old omnibus_updater cookbook. The modern https://github.com/chef-cookbooks/chef_client_updater cookbook is not supposed to have the issue.

Zendesk 18107

Have more questions? Submit a request

Comments

Powered by Zendesk