knife-ec-backup Node Export Failure

[sean-horn]

Using the knife-ec-backup 2.0.6 and chef_fs that ships with Chef Server 12.2.0, I am unable to do a successful knife-ec-backup of a system having 4 cookbooks, the _default environment, and a single node created with

knife node create kitchen-node -d -y

I have confirmed that the supplied example customer node also works fine after a workaround.

This issue may have the same root cause as #74 as the failure stack trace is identical. Everything goes swimmingly up to the point where knife-ec-backup attempts to export the node and write it to the local filesystem.

I'll show the issue first, then a workaround that seems to work for this limited case.

Issue

$ knife ec backup testbackup

....
lots of output of other objects
....

Created /nodes
DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_request
DEBUG: Signing the request as rainbowdash
DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_request
DEBUG: Initiating GET to https://127.0.0.1/organizations/4thcoffee/nodes
DEBUG: ---- HTTP Request Header Data: ----
DEBUG: Accept: application/json
DEBUG: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
DEBUG: X-OPS-SIGN: algorithm=sha1;version=1.0;
DEBUG: X-OPS-USERID: rainbowdash
DEBUG: X-OPS-TIMESTAMP: 2015-09-16T19:00:38Z
DEBUG: X-OPS-CONTENT-HASH: 2jmj7l5rSw0yVb/vlWAYkK/YBwk=
DEBUG: X-OPS-AUTHORIZATION-1: Esh3VkTl8L1c2GZkabk17W2PbV0uBzipzlAR5sQ1CCMi2Kwg2me6dMEK+FIP
DEBUG: X-OPS-AUTHORIZATION-2: DvUvx78QyNOOJvW777T1sHmrCxbGdc1Szdal1YTVDgU4Gj2Rjg6T4M1lsl31
DEBUG: X-OPS-AUTHORIZATION-3: s7HAk0MDPPXsh6YFwLUoO362VU+5LXGlxXNNtBLHDzYK5QRaCZAiwK7jIuVx
DEBUG: X-OPS-AUTHORIZATION-4: Qnn6eeZweDNhKAn+R8w4blbL0rMRIS06HmnWHL9oWjhr7MADCG/qzyfJzFdu
DEBUG: X-OPS-AUTHORIZATION-5: RFeNXlEvDAKOQMeg2Kbvoqp30h+3T3uIvqQjHoz6xrZAdxpI/EhT6MXZ9DBS
DEBUG: X-OPS-AUTHORIZATION-6: cDnhG7qijTikeK7MO0xpTMibmfl2JcWG4SygseJ9vA==
DEBUG: HOST: 127.0.0.1:443
DEBUG: X-Ops-Server-API-Version: 0
DEBUG: X-REMOTE-REQUEST-ID: b08b210c-f1ca-4a06-a149-4fdfe9061844
DEBUG: x-ops-request-source: web
DEBUG: ---- End HTTP Request Header Data ----
DEBUG: ---- HTTP Status and Header Data: ----
DEBUG: HTTP 1.1 200 OK
DEBUG: server: openresty/1.7.10.1
DEBUG: date: Wed, 16 Sep 2015 19:00:38 GMT
DEBUG: content-type: application/json
DEBUG: transfer-encoding: chunked
DEBUG: connection: close
DEBUG: x-ops-server-api-version: {"min_version":"0","max_version":"1","request_version":"0","response_version":"0"}
DEBUG: x-ops-api-info: flavor=cs;version=12.0.0;oc_erchef=12.2.0
DEBUG: content-encoding: gzip
DEBUG: ---- End HTTP Status/Header Data ----
DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_response
DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_response
DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_response
DEBUG: decompressing gzip response
DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_response
DEBUG: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_response
DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_response
DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_request
DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_request
DEBUG: Signing the request as rainbowdash
DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_request
DEBUG: Initiating GET to https://127.0.0.1/organizations/4thcoffee/nodes/kitchen_node
DEBUG: ---- HTTP Request Header Data: ----
DEBUG: Accept: application/json
DEBUG: Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3
DEBUG: X-OPS-SIGN: algorithm=sha1;version=1.0;
DEBUG: X-OPS-USERID: rainbowdash
DEBUG: X-OPS-TIMESTAMP: 2015-09-16T19:00:38Z
DEBUG: X-OPS-CONTENT-HASH: 2jmj7l5rSw0yVb/vlWAYkK/YBwk=
DEBUG: X-OPS-AUTHORIZATION-1: p13xUAkq5eL/DWqbhr+YDRSnVFdkMjpd+SAdwlwxRT2Smzo8UfIM4hjCU+qh
DEBUG: X-OPS-AUTHORIZATION-2: xAa65Y6ZNCUEq38/Mpt/pOPg5Uz5Ii5vR2nB0+SwSDeVb38VTi3hNTWKVUwi
DEBUG: X-OPS-AUTHORIZATION-3: fH5IFCk1+ckPkCO5kCe8Y0PzRVxzSMe8IK70MkxJKC30XEv79JJf+2Q+gtHl
DEBUG: X-OPS-AUTHORIZATION-4: MlVdByt0N8v3GmC/iMfoieJtmCcVVJcTQ05nrQIWM4U5I6Ls7iYRHzZBaFKs
DEBUG: X-OPS-AUTHORIZATION-5: JJzFL//jmF+0dA0beuN8l2B4vF+D1wfOY+Vlk4eon21k0OwfhAmXP5Fg+O0C
DEBUG: X-OPS-AUTHORIZATION-6: FQYa3sZk1nl/rqPEjWydp8JohuFk8+V3TZGgAB8Tfw==
DEBUG: HOST: 127.0.0.1:443
DEBUG: X-Ops-Server-API-Version: 0
DEBUG: X-REMOTE-REQUEST-ID: b08b210c-f1ca-4a06-a149-4fdfe9061844
DEBUG: x-ops-request-source: web
DEBUG: ---- End HTTP Request Header Data ----
DEBUG: ---- HTTP Status and Header Data: ----
DEBUG: HTTP 1.1 200 OK
DEBUG: server: openresty/1.7.10.1
DEBUG: date: Wed, 16 Sep 2015 19:00:38 GMT
DEBUG: content-type: application/json
DEBUG: transfer-encoding: chunked
DEBUG: connection: close
DEBUG: x-ops-server-api-version: {"min_version":"0","max_version":"1","request_version":"0","response_version":"0"}
DEBUG: x-ops-api-info: flavor=cs;version=12.0.0;oc_erchef=12.2.0
DEBUG: content-encoding: gzip
DEBUG: ---- End HTTP Status/Header Data ----
DEBUG: Chef::HTTP calling Chef::HTTP::RemoteRequestID#handle_response
DEBUG: Chef::HTTP calling Chef::HTTP::Authenticator#handle_response
DEBUG: Chef::HTTP calling Chef::HTTP::Decompressor#handle_response
DEBUG: decompressing gzip response
DEBUG: Chef::HTTP calling Chef::HTTP::CookieManager#handle_response
DEBUG: Chef::HTTP calling Chef::HTTP::JSONOutput#handle_response
DEBUG: Chef::HTTP calling Chef::HTTP::JSONInput#handle_response
Created /nodes/kitchen_node.json
/opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.0.current.0/lib/chef/chef_fs/file_system/file_system_entry.rb:86:in `exists?': undefined method `can_have_child?' for nil:NilClass (NoMethodError)
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.0.current.0/lib/chef/chef_fs/file_system.rb:411:in `get_or_create_parent'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.0.current.0/lib/chef/chef_fs/file_system.rb:144:in `block in copy_to'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.0.current.0/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:in `call'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.0.current.0/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:in `process_input'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.0.current.0/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:257:in `process_one'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.0.current.0/lib/chef/chef_fs/parallelizer.rb:93:in `call'
    from /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.0.current.0/lib/chef/chef_fs/parallelizer.rb:93:in `worker_loop'

Workaround

If I make the following change to the exists? method in /opt/opscode/embedded/lib/ruby/gems/2.1.0/gems/chef-12.5.0.current.0/lib/chef/chef_fs/file_system/file_system_entry.rb, the export completes successfully, not failing on the export of the single node, kitchen_node as shown above.

        def exists?
#          File.exists?(file_path) && parent.can_have_child?(name, dir?)
          File.exists?(file_path)
        end

[stevendanna]

@sean-horn Are you able to reproduce this with just cheffs? I.e. via knife download at all?

[sean-horn]

@stevendanna No, it seems to work fine with knife download /nodes/kitchen_node.json

I checked and I am using the original cheffs code, before my workaround to do the following

# knife download /nodes/kitchen_node.json
Created nodes
Created nodes/kitchen_node.json

# ls -al nodes
total 32
drwxr-xr-x 1 root root    34 Sep 16 20:16 .
drwxr-xr-x 1 root root    58 Sep 16 20:16 ..
-rw-r--r-- 1 root root 31968 Sep 16 20:16 kitchen_node.json

[pmoosh]

Encounter the same issue since updating to Chef server 12.2.
knife download seems to work.

[stevendanna]

Thanks for the confirmation. I'll take a look.

[pmoosh]

Just to make sure that is a issue with versions, I tested it with 2 different knife versions:
The error occurs with:
/opt/opscode/embedded/bin/knife --version
Chef: 12.5.0.current.0
knife-ec-backup works with:
/opt/chef/embedded/bin/knife --version
Chef: 12.4.1

[mikebthun]

How to roll back to version 12.4.1...

/opt/opscode/embedded/bin/gem uninstall -aIx chef chef-config ohai

/opt/opscode/embedded/bin/gem install chef --version=12.4.1
/opt/opscode/embedded/bin/gem install chef-config --version=12.4.1
/opt/opscode/embedded/bin/gem install ohai --version=8.5.1

[stevendanna]

I've found the root cause and am working on determining the right fix in ChefFS. This will be fixed in the next version of chef-server. Notably, this is failing on certain "root_files" in the backup (invitations.json, org.json, members.json) whose parent is the root of the directory and whose grandparent is nil (since in ChefFS the parent of the root directory is nil).

[stevendanna]

I believe this ChefFS fix will address the root cause. chef/chef#4091

[sdelano]

Now that the above PR is merged, how do we get it into knife-ec-backup and chef-server?

[stevendanna]

@sdelano I believe for chef-server, the latest nightlies should pick it up and use it automatically. Once there is a release of chef with that fix, I'll also pin the gem dependencies.

[scalp42]

@stevendanna looks like it shipped in 12.6.0.

[stevendanna]

@scalp42 Thanks for the remider. I've opened #77