If you ever have to run knife cookbook delete COOKBOOK_NAME --purge for whatever reason, the purge will wipe out any files with the same checksum that belong to other cookbooks.
When clients come along and ask for all the file contents, they will receive 404s for that purged file and their Chef Client run will fail during the cookbook syncing phase.
A way to recover from this condition is to drop the original contents in the correct location on disk after adding the appropriate bookshelf header to the cookbook file contents.
Bookshelf File Paths
Let's talk about the path of the files first. We can use the ruby digest/sha1 gem in an IRB session to generate the bookshelf paths under /var/opt/opscode/bookshelf/data/bookshelf if we know the name of the file. When a bookshelf file content download fails with a 404, but the cookbook exists, an error will be lodged in several of the files under /var/log/opscode/bookshelf. You can look in current and crash.log. This error will contain the name of the file in the format shown below where a name is assigned to the entry variable.
irb(main):001:0> require 'digest/sha1'
irb(main):003:0> entry = "organization-008dd329b94ec64374e24d68698eaad2%2Fchecksum-51d3f22a1d987eaf8a3ce0f4849c5b65"
irb(main):004:0> entry_hash = Digest::SHA1.hexdigest(entry)
Back on the system, I see that there are a ton of these directories available
[root@chef-432 bookshelf]# ls
00 06 0c 12 18 1e 24 2a 31 38 3e 45 4c 52 58 5e 64 6a 70 76 7c 82 88 8e 96 9c a2 a8 ae b4 ba c0 c6 cd d3 da e1 e7 ee f4 fa
01 07 0d 13 19 1f 25 2b 32 39 3f 47 4d 53 59 5f 65 6b 71 77 7d 83 89 90 97 9d a3 a9 af b5 bb c1 c7 ce d4 db e2 e8 ef f5 fb
02 08 0e 14 1a 20 26 2d 33 3a 40 48 4e 54 5a 60 66 6c 72 78 7e 84 8a 91 98 9e a4 aa b0 b6 bc c2 c8 cf d5 dd e3 e9 f0 f6 fc
03 09 0f 15 1b 21 27 2e 34 3b 41 49 4f 55 5b 61 67 6d 73 79 7f 85 8b 93 99 9f a5 ab b1 b7 bd c3 c9 d0 d7 de e4 eb f1 f7 fd
04 0a 10 16 1c 22 28 2f 35 3c 43 4a 50 56 5c 62 68 6e 74 7a 80 86 8c 94 9a a0 a6 ac b2 b8 be c4 ca d1 d8 df e5 ec f2 f8 fe
05 0b 11 17 1d 23 29 30 37 3d 44 4b 51 57 5d 63 69 6f 75 7b 81 87 8d 95 9b a1 a7 ad b3 b9 bf c5 cb d2 d9 e0 e6 ed f3 f9 ff
Ok, let's try to get to a file we know should be there. On my test system, this particular file is in the data storage area. If we are receiving a 404 for a cookbook file where the cookbook itself otherwise exists, then we would know to not expect it to be there and that this recovery process applies.
[root@chef-432 bookshelf]# ls 77/10/a4/4c/organization-008dd329b94ec64374e24d68698eaad2%2Fchecksum-51d3f22a1d987eaf8a3ce0f4849c5b65
Notice in the IRB code above that we generate a hash in the entry_hash variable based on the string that we save in the entry variable. This is just how the Chef Server erlang code views it too.
Then, we can notice the first eight characters in that entry_hash. In groups of two, they are 77 10 a4 4c. If you look closely at the final ls command above, you will see that my shell is in /var/opt/opscode/bookshelf/data/bookshelf already, and I've specified a path to the actual file content file using those groups of two, followed by the actual filename composed of the Chef Server organization in addition to a %2F (that's a forward slash encoded), as well as the name of the file based on its actual checksum.
Replace Purged Files
To replace 404s for purged cookbook files that should actually be there, take the attached script and proceed thusly:
- Retrieve the original content of the file from git or wherever. You can sometimes find that content in /var/chef/cache on a system that runs a Chef Client and has the cookbook with the missing file in its run list.
- Place that content in a file. We'll call it PLAIN_FILE here.
- Run the attached script on that file like this on your Chef Server. I would specify a temp path for the BOOKSHELF_FILE, then move it into place afterwards to the actual bookshelf data store location I explained above in "Bookshelf File Paths": /opt/opscode/embedded/escript/bookshelf-header-adder.escript PLAIN_FILE BOOKSHELF_FILE. This script serves to produce a new file which contains the necessary 2 bytes of bookshelf magic plus 16 bytes of digest plus newline as a header, followed by the plain text data that originally made up the cookbook file.
- Check the generated file contents, then move it into the correct path and filename under /var/opt/opscode/bookshelf/data/bookshelf using the "Bookshelf File Paths" information above.
- Test another download of the cookbook, either by clicking through to the file using Manage and noticing that it's not hanging on that file anymore (sometimes to find purged files, you must click a Download button where cookbook file content would normally be), or request the cookbook and version using knife raw /cookbooks/<name>/<version>
- In the output, you will see all the files that compose the cookbook, along with their bookshelf URLs. Take the bookshelf URL from one of them and paste it in a command like this
# curl -k 'https://chef-432.lxc:443/bookshelf/organization-008dd329b94ec64374e24d68698eaad2/checksum-51d3f22a1d987eaf8a3ce0f4849c5b65?AWSAccessKeyId=8c7e629ae6aab761d2f08a27cae4b6d19cd492f1&Expires=1523068658&Signature=M/23keYbnRZNfU8aVH912LnNaKo%3D'
# Cookbook:: test-nested-resource-a1
# Recipe:: default
# Copyright:: 2018, The Authors, All Rights Reserved. #chef_ingredient "chefdk" do
# action :install
# version :latest
# channel :stable
....more default recipe file content follows, shortening for readability
If you get to this point with no hangs or problems retrieving the targeted cookbook file, you have successfully replaced the file or files that were destroyed by the --purge