GNOME Bugzilla – Bug 765630
sha512sum is usually not installed on non-GNU systems
Last modified: 2019-07-29 19:27:37 UTC
sha512sum command is a part of GNU coreutils. On systems like *BSD, essential system commands are provided by the operating system itself and GNU coreutils is usually not installed. sha512sum command is not specified by the standard, so it doesn't get implemented on these systems. Most *BSD systems do provide their own command line tools to check SHA512 hash, but they have different syntax and input formats. I will attach a patch to allow using FreeBSD sha512 command and NetBSD/OpenBSD cksum command to check SHA512 hash. If none of them is found, use shasum command written in perl.
Created attachment 326795 [details] [review] build: Allow validating downloads with programs other than sha512sum On non-GNU systems, GNU coreutils are usually not installed, and commands like sha512sum are not available. This commit adds support for using builtin programs on FreeBSD, NetBSD, OpenBSD. Scripts added to configure.ac not only check whether an executables with a specific name is available but also verify whether the program works as expected. We are required to do so because different systems can include tools with the same name but using different command line syntaxes or input formats. If none of system-specific tools is found, we fallback to shasum program written in perl. This happens on DragonFlyBSD, which doesn't seem to have a builtin command line tool to compute SHA512 hash.
Would downgrading to sha256 improve the situation ? Is there really no one tool that's available on all systems and commonly installed? The patch is rather too involved, IMHO... Also, we already require GNU make and GCC, why is a coreutils req really a problem?
(In reply to Christian Persch from comment #2) > Would downgrading to sha256 improve the situation ? I think no, we will still have GNU sha256sum, FreeBSD/DragonFlyBSD sha256, NetBSD/OpenBSD cksum. GNU and FreeBSD/DragonFlyBSD also have cksum command, but their implementations can only do CRC, not SHA. > Is there really no one > tool that's available on all systems and commonly installed? I think 'openssl dgst -sha512' is probably the most commonly installed tool. It can do SHA hash, but I don't find an option that can read hash values from a file and check them. It is still usable, but we have to read the list and compare hash values with shell commands. > The patch is > rather too involved, IMHO... > > Also, we already require GNU make and GCC, why is a coreutils req really a > problem? When installing coreutils on *BSD, all commands are prefixed with 'g' to prevent conflict with existing system commands. cp becomes gcp, mkdir becomes gmkdir, install becomes ginstall. The problem is that GNU autotools prefer g-prefixed commands to non-prefixed commands. If coreutils is installed, configure script will pick gmkdir instead of mkdir, ginstall instead of install. This means we will not be able to check whether a project uses GNU-specific command options. The source code may build fine on our machines, but break in a clean chroot or on users' machines. I know it is possible to prevent GNU autotools from picking g-prefixed commands by setting environment variables like MKDIR_P and INSTALL, but I think it is better to keep the default working. GNU make is not a problem because the configure script doesn't automatically set MAKE to gmake. GCC is not required. We use clang to build gucharmap on FreeBSD. GCC requirement is not a problem for C or Fortran projects, but it is a big problem for C++ projects because mixing LLVM libc++ and GCC libstdc++ is very likely to cause crashes.
I could change the format of the unicode.sha512sum file to use the --tag switch of sha512sum which produces a BSD-compatible format of "SHA512 (Filename) = Hash" lines, that might make the task easier to adapt to BSDs? Thanks for the info about the g-prefixed programmes; I added gsha512sum to the programmes checked for, so it should at least work now if you do have the coreutils installed.
(In reply to Christian Persch from comment #4) > I could change the format of the unicode.sha512sum file to use the --tag > switch of sha512sum which produces a BSD-compatible format of "SHA512 > (Filename) = Hash" lines, that might make the task easier to adapt to BSDs? Unfortunately, I cannot find a builtin tool to read "SHA512 (Filename) = Hash" line on FreeBSD and DragonFlyBSD. Their builtin sha256 or sha512 tool does write output in this format, but it doesn't have the ability to use the format as input. I think the best solution may be including both coreutils and BSD format of hash lines. On GNU, we can use coreutils with either coreutils or BSD format. On FreeBSD and DragonFlyBSD, we can use shasum from perl with coreutils format. On NetBSD and OpenBSD, we can use the builtin cksum tool with BSD format. This will keep commands used to check hashes simple because no format conversion is required, but we still need to check whether cksum command works as expected to avoid using cksum commands on GNU, FreeBSD, DragonFlyBSD.
Created attachment 331748 [details] [review] build: Allow validating downloads with programs other than sha512sum On non-GNU systems, GNU coreutils are usually not installed, and commands like sha512sum are not available. This commit adds support for using shasum program written in perl and builtin programs on NetBSD and OpenBSD. Scripts added to configure.ac not only check whether an executables with a specific name is available but also verify whether the program works as expected. We are required to do so because different systems can include tools with the same name but using different command line syntaxes or input formats.
This version reduces the number of 'if' from 4 to 2, and it avoids doing format conversion in Makefile.am by including both GNU and BSD-style sha512sum.
The new patch has stayed in bugzilla for one month. Can it be reviewed now?
Created attachment 336272 [details] [review] build: Allow validating downloads with programs other than sha512sum On non-GNU systems, GNU coreutils are usually not installed, and commands like sha512sum are not available. This commit adds support for using shasum program written in perl and builtin programs on NetBSD and OpenBSD. Scripts added to configure.ac not only check whether an executables with a specific name is available but also verify whether the program works as expected. We are required to do so because different systems can include tools with the same name but using different command line syntaxes or input formats.
Created attachment 353234 [details] [review] build: Allow validating downloads with programs other than sha512sum On non-GNU systems, GNU coreutils are usually not installed, and commands like sha512sum are not available. This commit adds support for using shasum program written in perl and builtin programs on NetBSD and OpenBSD. Scripts added to configure.ac not only check whether an executables with a specific name is available but also verify whether the program works as expected. We are required to do so because different systems can include tools with the same name but using different command line syntaxes or input formats. The previous patch only checks for validation programs when --with-unicode-data is set to 'no', which is an error because 'no' is never an acceptable value of --with-unicode-data option. It causes the checking to be skipped and validation scripts in Makefile to be commented out without showing any error messages. This updated patch fixes the problem mentioned above.
One small comment from an OpenBSD dude here :-) The only problem with that logic is that if gsha512sum is found at configure time, it will be used for building. While that works fine for manual builds, in a full package bulk build, there is no guarantee that gsha512sum will not be removed in the middle of the build unless it is an explicit dependency; which is exactly what we are trying to prevent. Maybe reverse the checks? (look for cksum first, then sha512sum) I guess I can always set CKSUM but I wanted to mention this anyway. Thanks.
Created attachment 353257 [details] [review] build: Allow validating downloads with programs other than sha512sum On non-GNU systems, GNU coreutils are usually not installed, and commands like sha512sum are not available. This commit adds support for using shasum program written in perl and builtin programs on NetBSD and OpenBSD. Scripts added to configure.ac not only check whether an executables with a specific name is available but also verify whether the program works as expected. We are required to do so because different systems can include tools with the same name but using different command line syntaxes or input formats. I updated the patch to check cksum before sha512sum. I also changed the order of cksum and sha512sum in other places to make the patch look consistent. I still don't understand why the order matters in bulk package building ... Most package builders disable internet access during the build so --with-unicode-data=download is not usable. This means cksum or sha512sum will never be used even if configure script checks for them.
> I still don't understand why the order matters in bulk package building ... > Most package builders disable internet access during the build so > --with-unicode-data=download is not usable. This means cksum or sha512sum > will > never be used even if configure script checks for them. That's correct but I'd rather not rely on that. It's just a precaution.
Created attachment 354316 [details] [review] build: Allow validating downloads with programs other than sha512sum On non-GNU systems, GNU coreutils are usually not installed, and commands like sha512sum are not available. This commit adds support for using shasum program written in perl and builtin programs on NetBSD and OpenBSD. Scripts added to configure.ac not only check whether an executables with a specific name is available but also verify whether the program works as expected. We are required to do so because different systems can include tools with the same name but using different command line syntaxes or input formats.
Created attachment 361603 [details] [review] build: Allow validating downloads with programs other than sha512sum On non-GNU systems, GNU coreutils are usually not installed, and commands like sha512sum are not available. This commit adds support for using shasum program written in perl and builtin programs on NetBSD and OpenBSD. Scripts added to configure.ac not only check whether an executables with a specific name is available but also verify whether the program works as expected. We are required to do so because different systems can include tools with the same name but using different command line syntaxes or input formats.
Ping ... Will this get reviewed?
Created attachment 372706 [details] [review] build: Allow validating downloads with programs other than sha512sum On non-GNU systems, GNU coreutils are usually not installed, and commands like sha512sum are not available. This commit adds support for using shasum program written in perl and builtin programs on NetBSD and OpenBSD. Scripts added to configure.ac not only check whether an executables with a specific name is available but also verify whether the program works as expected. We are required to do so because different systems can include tools with the same name but using different command line syntaxes or input formats.
Ping ... Should I copy this patch to GitLab in order to get it reviewed?
This patch is obsolete since with the meson port done, you always have to supply the path to the unicode data files; automatic downloading is no longer an option.