xz: Don't fail if Capsicum is enabled but kernel doesn't support it.

(This commit combines related commits from the master branch.)

If Capsicum support is missing from the kernel or xz is being run
in an emulator that lacks Capsicum suport, the syscalls will fail
and set errno to ENOSYS. Previously xz would display and error and
exit, making xz unusable. Now it will check for ENOSYS and run
without sandbox support. Other tools like ssh behave similarly.

Displaying a warning for missing Capsicum support was considered
but such extra output would quickly become annoying. It would also
break test_scripts.sh in "make check".

Also move cap_enter() to be the first step instead of the last one.
This matches the example in the cap_rights_limit(2) man page. With
the current code it shouldn't make any practical difference though.

Thanks to Xin Li for the bug report, suggesting a fix, and testing:
https://github.com/tukaani-project/xz/pull/43

Thanks to Jia Tan for most of the original commits.

src/xz/file_io.c

@@ -192,6 +192,9 @@ io_sandbox_enter(int src_fd)
 	// Capsicum needs FreeBSD 10.0 or later.
 	cap_rights_t rights;
 
+	if (cap_enter())
+		goto error;
+
 	if (cap_rights_limit(src_fd, cap_rights_init(&rights,
 			CAP_EVENT, CAP_FCNTL, CAP_LOOKUP, CAP_READ, CAP_SEEK)))
 		goto error;
@@ -209,9 +212,6 @@ io_sandbox_enter(int src_fd)
 			CAP_WRITE)))
 		goto error;
 
-	if (cap_enter())
-		goto error;
-
 #elif defined(HAVE_PLEDGE)
 	// pledge() was introduced in OpenBSD 5.9.
 	//
@@ -232,6 +232,15 @@ io_sandbox_enter(int src_fd)
 	return;
 
 error:
+#ifdef HAVE_CAPSICUM
+	// If a kernel is configured without capability mode support or
+	// used in an emulator that does not implement the capability
+	// system calls, then the Capsicum system calls will fail and set
+	// errno to ENOSYS. In that case xz will silently run without
+	// the sandbox.
+	if (errno == ENOSYS)
+		return;
+#endif
 	message_fatal(_("Failed to enable the sandbox"));
 }
 #endif // ENABLE_SANDBOX