GNOME Bugzilla – Bug 771244
gparted does not recognize the iso9660 file system in cloned Ubuntu USB boot drives
Last modified: 2019-07-21 14:41:28 UTC
As reported in (https://bugs.launchpad.net/ubuntu/+source/gparted/+bug/1622313) by sudodus: I am testing in a live system and looking at the very drive, from which it is booted, the current daily iso file of Lubuntu Xenial i386 (post 16.04.1 LTS). The problem is that the file system cannot be identified, and several end users may (and will) think that the USB boot drive is damaged. But it works, it is cloned, which is the straightforward method to create a boot drive from a hybrid iso file. this issue was worse in previous versions, where gparted would complain about an error; now it is at least seeing the partition. The next step is that it can see the file system too. lsblk can see it, as illustrated by the following command line (in a wide terminal window), sudo lsblk -fm The attached screenshot illustrates the problem, and shows that it affects the text mode program parted too. ProblemType: Bug DistroRelease: Ubuntu 16.04 Package: gparted 0.25.0-1 ProcVersionSignature: Ubuntu 4.4.0-38.57-generic 4.4.19 Uname: Linux 4.4.0-38-generic i686 ApportVersion: 2.20.1-0ubuntu2.1 Architecture: i386 CasperVersion: 1.376 CurrentDesktop: LXDE Date: Sun Sep 11 10:10:21 2016 LiveMediaBuild: Lubuntu 16.04.1 LTS "Xenial Xerus" - Beta i386 (20160909) ProcEnviron: TERM=xterm PATH=(custom, no user) XDG_RUNTIME_DIR=<set> LANG=en_US.UTF-8 SHELL=/bin/bash SourcePackage: gparted UpgradeStatus: No upgrade log present (probably fresh install) Edit: After some testing I found that the current i386 versions behave in a similar way as Xenial. But the current amd64 (64-bit) versions are more severely affected by this bug. It is much worse in Trusty, but also bad in Xenial as illustrated by screenshots in comments #6 and #7. This behaviour will really confuse new users of Ubuntu and the Ubuntu flavours. This illustrates that lsblk can see the iso9660 file system: lubuntu@lubuntu:~$ sudo lsblk -f /dev/sda NAME FSTYPE LABEL UUID MOUNTPOINT sda iso9660 Lubuntu 16.04.1 LTS i386 2016-09-09-18-15-41-00 /cdrom └─sda1 iso9660 Lubuntu 16.04.1 LTS i386 2016-09-09-18-15-41-00 lubuntu@lubuntu:~$ sudo lsblk -m /dev/sda NAME SIZE OWNER GROUP MODE sda 3.7G root disk brw-rw---- └─sda1 859M root disk brw-rw---- lubuntu@lubuntu:~$ The attached screenshot illustrates the worse problem in some older versions. It seems to work better in the i386 version of 14.04.1 (like in 16.04.1) but this screenshot was made in a live system booted from ubuntu-14.04.1-desktop-amd64.iso (https://launchpadlibrarian.net/283754813/screenshot-ubuntu-14.04.1-desktop-amd64-with-gparted.png) Having found this problem with the Trusty amd64 version, I tried the Xenial amd64 version too, and it is affected in a minor way. It shows the file system, but it gets the sector size wrong and hence the aize of the drive. My 32 GB seems to contain 119 GiB (128 GiB is reported by parted, and it thinks it is a mac file system). The the screenshots in this and the following comment. (https://launchpadlibrarian.net/283757272/screenshot-ubuntu-16.04.1-desktop-amd64-with-gparted-1.png) Screenshot #2 for Xenial amd64 ... (https://launchpadlibrarian.net/283757411/screenshot-ubuntu-16.04.1-desktop-amd64-with-gparted-2.png)
See also the behaviour in i386 (32-bit) systems as illustrated by the attached screenshot of Lubuntu 16.04.1 from the current daily iso file, screenshot-lubuntu-desktop-xenial-i386-with-gparted.png
Created attachment 335303 [details] The screenshot from the original report at Launchpad (32-bit Lubuntu)
Created attachment 337096 [details] [review] Add iso9660 file system detection (v1) Thank you Alberto and sudodus for your interest in improving GParted. Attached is a patch that adds iso9660 file system detection. To prepare a test I created a 6 GiB partition and then used *dd* to copy the contents of a kubuntu CD/DVD image to the partition. Please let me know if you are able to test the patch, and if so if it works for you. Curtis
Thank you. How could we test this patch?
Hi Alberto, To test the patch requires setting up a development environment on your GNU/Linux computer, cloning the gparted git repository, applying the patch, then compiling and testing that GParted recognizes the iso image you have cloned to a partition. See Developing GParted using Git [1]. [1] http://gparted.org/git.php If you are not comfortable with that, then you could wait until Mike Fleetwood has a chance to test. If the patch passes his review then it would get committed to the GParted git repository for inclusion in a future release of GParted and also the GParted Live image. Curtis
Created attachment 337218 [details] New grey colour palette Hi Curtis, After review I have a couple of questions. 1) Silver colour choice Is the iso9660 silver colour too close to the unallocate medium grey? Attached is a montage of GParted screen shots on CentOS 7 and Fedora 24 showing the current grey palette we have. Could switch to the GNOME 2 grey palette [1] for grub2 core, iso9660 and unallocated (from shadow to medium) but that involves changing the long standing shade of grey used for unallocated some what. Or try a to find a grey for iso9660 with greater separation from unallocated colour. [1] GNOME 2, 32-colour palette https://developer.gnome.org/hig-book/2.32/design-color.html.en 2) libparted error Using one of the iso9660 test images from util-linux, on a whole disk causes, libparted to throw an exception and therefore for GParted to display this error dialog: Libparted Error (-) Invalid partition table - recursive partition on /dev/md126. [ Cancel ] [ Ignore ] wget http://git.kernel.org/cgit/utils/util-linux/util-linux.git/plain/tests/ts/isosize/sample.iso.gz gunzip sample.iso.gz dd if=sample.iso of=/dev/md126 blkid /dev/md126 parted /dev/md126 print gparted /dev/md126 Blkid happily recognised sample.iso, but parted/libparted doesn't like it. The other iso9660 images used for recognition testing by blkid [2] don't cause the exception. Might well choose to ignore this even though blkid happily recognises sample.iso in a whole disk device. [2] iso.img.xz, iso-joliet.img.xz & iso-rr-joliet.img.xz http://git.kernel.org/cgit/utils/util-linux/util-linux.git/tree/tests/ts/blkid/images-fs Thanks, Mike
_Intro_ It is great that you are improving gparted :-) I am developing and testing software at the user's level (including script-based software like 'my' mkusb). Unfortunately I am not advanced enough to compile gparted from the source code, but if you can provide a package for a major distro, I can install and test it, for example a deb package for debian and ubuntu. That way can will get help from me with extensive testing. _Background_ What Alberto and I want is a fix so that gparted can recognize typical *isohybrid* partition tables with the 'iso9660' file system and the special structure, that makes such iso files possible to clone to bootable CD/DVD disks as well as to bootable USB pendrives and flash cards. Maybe we should also make it recognize 'udf' file systems with the same or similar isohybrid partition tables. This kind of cloning (for example with dd or better with a tool, that wraps a 'safety belt' around the process) is the recommended method to create USB boot drives for many linux distros. gparted is a very good tool, but this bug makes people give up with their pendrives. Too many people think that the cloning process caused a damage, so that their pendrive will not work, and/or will not be possible to restore to standard mass storage devices with an MSDOS partition table and a FAT32 file system. _Test with mini.iso files (and other iso files of major linux distros_ So please test with some typical USB pendrives made by cloning iso files. When gparted does not complain, but sees the file system, the end users feel confident with their pendrive. If you cannot send me a compiled package, I suggest that you test the new version of gparted with some of the Debian or Ubuntu 'mini.iso' files (32-bit and 64-bit), which are small so quick to clone to a USB pendrive, as well as iso files of other major linux distrod (for example Arch, Fedora, Mageia, Manjaro, openSUSE). Many full size iso files (1-1.5) GB take a minute to clone to a fast USB pendrive. A slow pendrive may have a write speed of 5 MB/s, so it would need 3-6 minutes. _Comparison with lsblk_ This is how lsblk sees a USB boot drive made from a Debian mini.iso file. $ sudo lsblk -f /dev/sdd NAME FSTYPE LABEL UUID MOUNTPOINT sdd iso9660 ISOIMAGE 2016-09-13-22-21-51-00 ├─sdd1 iso9660 ISOIMAGE 2016-09-13-22-21-51-00 └─sdd2 vfat Firmware C000-DAC4 $ sudo lsblk -m /dev/sdd NAME SIZE OWNER GROUP MODE sdd 29,8G root disk brw-rw---- ├─sdd1 22M root disk brw-rw---- └─sdd2 6M root disk brw-rw---- gparted version 0.25.0 is not happy with it and dumps the following text (to a terminal window) ====================== libparted : 3.2 ====================== Invalid partition table - recursive partition on /dev/sdd. Invalid partition table - recursive partition on /dev/sdd. and the corresponding complaint and lack of output for the file system in gparted's GUI. --- _Report the bug upstream?_ If the main bug or missing feature to recognize the isohybrid partition table structure, maybe we should create a bug report against libparted. Or you should use the same method as lsblk (and/or blkid) for gparted to recognize these kinds of partition tables and file systems.
Testing this is something I could do, but I have never done. So I had to discover how to do it, and even it looks simple I'm currently out of time. I will take note of this bug in my todo list, and if nobody else has verified the patch when I have finished the more important tasks I will take the job myself.
Thank you Alberto and sudodus for your comments. No worries if you are unable to test. We are all time-constrained which is why we ask for help with testing. We also understand when there is a lack of time for additional help. We do our best with the precious free time we have. Regarding packages, these are built by others that maintain the software packages for the upstream GNU/Linux distributions. We do not build the packages. @sudodus, Would you consider opening an upstream bug report [1] for the libparted error? [1] https://www.gnu.org/software/parted/lists.shtml I do not anticipate a fix anytime soon, but at least you would raise the issue in case someone has time and energy to address it. Regarding changing GParted's method of recognizing partitions from libparted to lsblk, I think the task is non-trivial. GParted is based upon libparted and uses libparted both for recognition and manipulation of partitions. Changing this foundation would most likely introduce many different issues. @Mike, Thank you for the patch review. A1) Regarding colours, I agree that silver is close to other grey colours. I tend to think of shades of grey as representing unsupported file systems or unallocated space. I consider silver a shade of grey, and I can try picking a different shade. However, I do think that we will inevitably run into situations where the colours are close in shade or hue. If you have a preferred colour code in mind then I'm all ears. :-) A2) Regarding the libparted error, I think we may just have to ignore it for now. I thought about trying to make GParted ignore this particular error about recursive partitions, but then thought better of it. My reasoning is that it is possible that there are other cases where recursive partitions are used outside of an ISO image and I would not want to silence the libparted error. A downside of not silencing the libparted error is that GParted will only recognize ISO images if no error is encountered. In other words when recursive partitions are encountered, then the error will take precedence. This is no different than the current behaviour. I will draft a new patch. Curtis
Yes I will consider opening an upstream bug report [1] for the libparted error [1] https://www.gnu.org/software/parted/lists.shtml I can't do it right now, but it is on my to-do list :-) -o- I hope that you can make an exception for the recursive partition structure typical for linux iso files. Otherwise it is a good idea to let gparted complain, when it sees such monsters.
Created attachment 337247 [details] GParted gray colour palette 2 Hi Mike, What do you think of using light gray (#D3D3D3)? The attached screen shot shows the new light gray on the alternating line colours we have in the text partition lest. Curtis
Created attachment 337249 [details] [review] Add iso9660 file system detection (v2) I have updated the patch to include the new light gray colour for iso9660 images. In the patch I moved FS_LUKS to more closely match the order in include/Utils.h. Curtis
Obviously postponing this wasn't a good idea on my side. I will test it shortly.
Created attachment 342492 [details] [review] Add iso9660 file system detection (v3) Hi Curtis, Finally had some time to spend enhancing the fix for this bug report. Here is patchset v3. Builds on top of your v2 patch from comment 12. Additionally does: 1) Avoids calling libparted to read the partition table when there is a recognised whole disk device file system (I.E. iso9660). This avoids generating various libparted errors and warning. The ones I found during development with the ISO images mentioned in the commit messages and various libparted versions were: * Libparted Error (-) Invalid partition table - recursive partition on /dev/sdc. [ Cancel ] [ Ignore ] * Libparted Warning /!\ /dev/sdc contains GPT signatures, indicating that it has a GPT table. However, it does not have a valid fake msdos partition table, as it should. Perhaps it was corrupted -- possibly by a program that doesn't understand GPT partition tables. Or perhaps you deleted the GPT table, and are now using an msdos partition table. Is this a GPT partition table? [ Yes ] [ No ] * Libparted Warning /!\ The drive descriptor says the physical block size is 2048 bytes, but Linux says it is 512 bytes. [ Cancel ] [ Ignore ] 2) Ensures that there are always blkid cache entries (even if they are blank ones) for whole disk devices. This is so whole disk device iso9660 detection works on recent distros which detect partitions inside iso9660 images and blkid doesn't report the iso9660 on the whole disk device. Thanks, Mike
Created attachment 342709 [details] [review] Add iso9660 file system detection (v4) Hi Curtis, Here's patchset v4. Compared to v3 there is just a couple of minor corrections in the last commit message. Mike
Hi Mike, Thank you for the enhancements to this patch set. From my testing all appears to work as desired with no errant error messages. Prior to committing patch set v4 from comment #15 I made one minor correction in the comment portion of 3 of 4. Fix minor typo. CHANGE FROM: # ./gpartedbun /dev/sdc ^^^ TO: # ./gpartedbin /dev/sdc This patch set has been committed to the git repository for inclusion in the next release of GParted. The relevant git commits can be viewed at the following links: Add detection of iso9660 file system (#771244) https://git.gnome.org/browse/gparted/commit/?id=f762c0cc34f5d414bce6b29179bd7487baa33a10 Create new method GParted_Core::set_device_from_disk() (#771244) https://git.gnome.org/browse/gparted/commit/?id=fd426cf36e2bfa1f2e21cb52ed396f8f7afe01ef Only read partition table after not finding a whole disk file system (#771244) https://git.gnome.org/browse/gparted/commit/?id=683dc9d1abd6a388e6f1eda7c7ba8465eb970756 Ensure blkid FS_Info cache has entries for all whole disk devices (#771244) https://git.gnome.org/browse/gparted/commit/?id=b2190372d04f09bd38a7395b38e43034f1733d81 Curtis
This enhancement was included in the GParted 0.28.0 release on February 14, 2017.
Hi! I hope this is the correct ticket. As opposed to wipefs, I like to just replace the 0x01 byte (type = Primary Volume Descriptor) in front of the CD001 with something invalid like 0x0A so I can easily find it again in my hex editor to toggle it. gparted still shows the disk as ISO 9660. To my understanding, a Primary Volume Descriptor is required for ISO 9660 compliance, so I thing gparted should report this as something like "unknown, similar to ISO 9660" with lowest priority, and prefer any valid alternatives available (in my case, the GPT).