GNOME Bugzilla – Bug 575324
Copying a huge NTFS partition (e.g., > 100 GB) never finishes
Last modified: 2010-03-01 21:02:23 UTC
Please describe the problem: When using the UI to copy the partition, it runs for a long time and then stops all disk activity, ntfsclone is nowhere evident in the task list or top but the UI never moves on to the next step in the task. Steps to reproduce: Try to copy a huge NTFS partition (>750 GB) to another disk > 1TB using an MS-Dos partition table. Actual results: It never finishes after the ntfsclone step; the UI refuses to continue (no system issues, no crash, no UI freeze - it can be exited and all the menus work). Expected results: The partition is copied correctly and the application returns to ready state. Does this happen every time? Yes Other information: This is an MS-DOS Partition table. I am running 32bit XP and cannot use GPT, even if I wanted to, which I shouldn't have to. This can't be the 2TB limit because I am only copying 750 GB and the target disk is 1.5TB only. The only way I got this critical task of mine to work (I NEEDED to copy this partition!) was to look at the code, and then copy a much smaller partition which did finish. Then I was able to duplicate the commands on the command line and I was able to finish it. Then, going back into the UI to grow the partition to the full 1.5 TB worked. This was the same whether it was with the latest version of the LiveCD (0.4.3-2) or Parted Magic running the latest version of gParted (0.4.3). I even tried an earlier version of the LiveCD (0.4.1-2) but same results. I saw somewhere in the forum that someone had reported this but didn't create a bug and the developer didn't acknowledge this. Rest assured, this is a real bug. A blocker for NTFS users with huge disks.
Thank you James for reporting this bug and providing a detailed description. After you could not find ntfsclone running and shut down gparted, do you recall if there was a copy of the partition on the new 1.5 TB disk? What I am trying to discover is if ntfsclone completed successfully or not when run from within the gparted application. Could you provide the output from running the following command in a terminal window? fdisk -l -u
Yes, sorry I forgot that. After all the times that this happened, there was, in fact a copy of the partition on the new disk. However, since I was not sure of the state of that partition and did not want to spend another number of hours doing a compare, I can't be sure if the partition was viable or not. It seemed to be but I didn't want to risk it. Like I said before, after a few hours when it looked to have completed, ntfsclone was not evident in the 'ps agx' list or 'top', so it looked like it completed successfully (?), but gparted did not see it for some strange reason and could not continue with its validation of the new partition and update of the partition table as normal. As for the 'fdisk -l -u', be advised that I needed to have this partition working, so I did this manually, as I said before, and finished the process via the command line (including writing to the partition table with dd as gparted would have done if this bug was not there.) So, I now have a working disk and partition and am running XP Pro, no problem. Would the output of 'fdisk -l -u' really be of any use to you from this? I can do that if you want. This involves booting up from the gparted disk again and running it. What I _can't_ do is attempt the partition copy again just so I can give you the output since I have been running on it a few days and have new data that I don't want to lose. This looks like a strange one, because I could not tell from the code that anything would cause gparted to hang on the ntfsclone invocation unless ntfsclone is not returning the correct value upon completion or never really completes but hangs for some reason but hidden from the process table?
Created attachment 130699 [details] Detail log from successfully coping an NTFS partition Attached for reference is a gparted details log from successfully copying a partition. When gparted invokes external commands such as ntfsclone, it spawns a shell that executes the command. When the command finishes, it passes an exit code to the shell. Then the shell finishes and passes the exit code to gparted. Hence if GParted was reporting that it was running the ntfsclone command, and ntfsclone did not show up in the process list, then why would the shell not have finished and returned an exit code to gparted? This is strange. Unfortunately I do not enough drive space to test copying large NTFS partitions of the size you indicated. ;-( To figure out what went wrong is difficult without being able to reproduce the problem. The reason I asked for an "fdisk -l -u" listing is that this would show the exact sector boundaries of the partitions. This may or may not be relevant, but is additional information that could be collected and might help lead to a solution.
Hello I can confirm this bug. It is happening to me with the copy of a 230G ntfs partition. I can reproduce the problem 1 or 2 times for you but please ask me quickly as I'll soon need to use the disk (and will therefore use another way). Here are first some infos I could gather while the operation was finished but gparted still waiting : The destination disk is a fresh new 500G disk on which I first made backups of another computer (the last 2 partitions : sdc1 and sdc2). I've created them at the beginning of the disk then I moved them to the end to give room for the next one at the beginning of the disk. The operation "simply" consists in copying a whole disk (sdb1, its only 230G partition) to the beginning of the new one (the 500G one). Both disks are plugged in my computer via USB 2. I'm using gparted 0.4.5-2ubuntu1 (karmic). I've previously made the same operation with smaller partitions (10G and 100G as seen in the following outputs) and it worked really fine, so the problem *seems* to live between 100G-230G for me. Some screenshots : http://img248.imageshack.us/img248/6534/screenshotapplyingpendi.png (main, background window) http://img163.imageshack.us/img163/9922/destinationdisk.png (destination disk, as the filename indicates) No ntfsclone in the process list. But gparted is still using 85% of the CPU (result of the "top" command) : PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26650 root 20 0 74488 47m 13m S 87.5 2.4 538:02.61 gpartedbin 1247 root 20 0 55668 18m 9.9m R 7.3 0.9 236:35.88 Xorg ... Here is the output of "fdisk -l -u" for the concerned disks : root@fondcombe:~# fdisk -l -u Disk /dev/sdb: 250.1 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x000628d9 Device Boot Start End Blocks Id System /dev/sdb1 63 488392064 244196001 7 HPFS/NTFS Disk /dev/sdc: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x44fdfe06 Device Boot Start End Blocks Id System /dev/sdc1 737801190 758284064 10241437+ 7 HPFS/NTFS /dev/sdc2 758284065 976768064 109242000 7 HPFS/NTFS /dev/sdc3 63 488392064 244196001 7 HPFS/NTFS Partition table entries are not in disk order
Created attachment 151791 [details] gparted waiting at the end of the operation
Created attachment 151792 [details] destination disk shows the new partition in the background
Thank you nicobo for adding to this bug report. Following are some requests: 1) Would you be able to provide a complete process listing after this problem has occurred? For example: $ ps -ef I am curious as to what other processes may still be outstanding, even in the ntfsclone process has disappeared. 2) Does the copy of the NTFS file system seem to be OK? Can you mount the file system, view your files, etc? 3) Would you be able to try the ntfsclone from the command line? For example using the command from your attached picture: $ ntfsclone -f --overwrite /dev/sdc3 /dev/sdb1 After the command completes also execute: $ echo $? This will display the return code from the previous command.
1) see new attachment (was done in the same X session, but after having closed gparted ; I can re-do it after the next try if needed) 2) Yes, the copy *seems* ok and I can mount the partition and access my files. By the way there are way too many files for me to check if everything is ok by hand and I don't know of a quick way to do it... 3) I come back to you asap
Created attachment 151794 [details] output of ps -ef after operation cancelled and gparted closed
1) If possible would you be able to retry the copy? Then while GParted is still up and running could you snag a complete process listing "ps -ef"? I am curious to see if there is a GParted spawned shell process still outstanding.
3) partition sdc3 deleted via gparted ; new empty partition created via fdisk (new part. starting at 1 and with exact same size as the source, then type set to 7 HPFS/NTFS) ; then : root@fondcombe:~# fdisk -l -u Disk /dev/sdb: 250.1 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x000628d9 Device Boot Start End Blocks Id System /dev/sdb1 63 488392064 244196001 7 HPFS/NTFS Disk /dev/sdc: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x44fdfe06 Device Boot Start End Blocks Id System /dev/sdc1 737801190 758284064 10241437+ 7 HPFS/NTFS /dev/sdc2 758284065 976768064 109242000 7 HPFS/NTFS /dev/sdc3 63 488392064 244196001 7 HPFS/NTFS Partition table entries are not in disk order root@fondcombe:~# ntfsclone -f --overwrite /dev/sdc3 /dev/sdb1 ntfsclone v2.0.0 (libntfs 10:0:0) NTFS volume version: 3.1 Cluster size : 4096 bytes Current volume size: 250056699904 bytes (250057 MB) Current device size: 250056705024 bytes (250057 MB) Scanning volume ... 100.00 percent completed Accounting clusters ... Space in use : 247608 MB (99.0%) Cloning NTFS ... 100.00 percent completed Syncing ... root@fondcombe:~# echo $? 0 root@fondcombe:~# fdisk -l -u Disk /dev/sdb: 250.1 GB, 250059350016 bytes 255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x000628d9 Device Boot Start End Blocks Id System /dev/sdb1 63 488392064 244196001 7 HPFS/NTFS Disk /dev/sdc: 500.1 GB, 500107862016 bytes 255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors Units = sectors of 1 * 512 = 512 bytes Disk identifier: 0x44fdfe06 Device Boot Start End Blocks Id System /dev/sdc1 737801190 758284064 10241437+ 7 HPFS/NTFS /dev/sdc2 758284065 976768064 109242000 7 HPFS/NTFS /dev/sdc3 63 488392064 244196001 7 HPFS/NTFS Partition table entries are not in disk order
From comment #11 I see that the ntfsclone command did complete successfully. Since GParted spawns a shell to execute commands, I wonder if the shell is still lingering around when this problem occurs? It would be appreciated if you could retry the copy using GParted and provide the output from a "ps -ef" before closing GParted.
Created attachment 151922 [details] before starting the cloning
Created attachment 151923 [details] during the cloning (with disk activity)
Created attachment 151924 [details] after the cloning (gparted still waiting)
Created attachment 151925 [details] after the cloning (gparted closed)
Created attachment 151926 [details] gparted log
Hi 1) I've attached the "ps -ef" results as 4 files : before, during, waiting and after the operation ; plus the detailed gparted log of the whole operation Hope this helps
Thank you nicobo for providing all of these various process listings. Both the ntfsclone process, and the shell from which it was spawned have disappeared from the process list. Meanwhile gparted is still waiting for the spawned shell to close. This is very strange behaviour indeed. At the moment I cannot think of any other requests to ask of you to narrow down this bug. I realize that you most likely need to get on with using your system so please feel free to do so. Thanks again for adding more information to this bug. I will have to ponder this a while to think of what the next step might be.
Any news on this? I can reproduce this bug by copying a 120GB NTFS-partition from a 640GB disk to a 1TB disk (copying a 50GB partition works fine). Once the I/O-activity stops, the RAM usages goes down, CPU usage increases, and according to the system monitor gparted is in a sleeping state (waiting channel: hrtimer_nanosleep). I used ntfscmp to check the copy and it seems complete. I'm using gparted 0.5.1 with Ubuntu 10.04 Alpha 3.
So far I have been unable to replicate the problem. I have two 160 GB drives. Today I tried creating a 120 GB NTFS partition on one, and then using GParted to copy from one to the other. The operation went smoothly and was successful. How much of your NTFS partition is in use?
The partition is pretty much full (120/126GB used), as I resized it before backing it up to the 640GB disk. Maybe worth mentioning: gpartedbin keeps hogging CPU even after cancelling the supposedly unfinished operation.
*** Bug 611284 has been marked as a duplicate of this bug. ***
FWIW, copying an empty 126GB partition works, while copying the same partition with 10GB data (created some files using dd if=/dev/zero) triggers the bug.
Heh, it took almost a year, but I am now finally getting confirmation of this bug from others. Oh well. I guess being an early adopter of technology (huge - >1TB disks) sometimes doesn't pay. As I said before in my previous expositions about this bug, it looks like parted itself is not returning a proper status or perhaps any status at all when it exits, causing the outer shell of gparted to continue waiting forever? Can this bug at least be now marked as "confirmed"? >1TB disks are now pretty cheap, so testing by the developer should be able to be done now.
Sorry for the misinformation: copying the partly filled 126GB partition to the 1TB disk did work - it just took more time than I thought (while showing the same "resource" symptoms as the bug). This brings me to the question, if the operation really wouldn't finish or if it would just take abnormal long? I let it run no longer than an hour once the ntfsclone process was gone before cancelling the operation. Think I'll let it run longer now.
I let it run for quite a few hours, a few times, without it finishing.
Attached gdb to gpartedbin: it seems to hang in the cleanup_cursor method of Utils.cc. The value of 'str' for IA__g_utf8_pointer_to_offset changes constantly (the percentage value of the string jumps around - it doesn't increase or decrease).
Created attachment 154907 [details] backtrace while hanging
Created attachment 154909 [details] full backtrace, note frame 2 of thread 2
FYI, I ran the test at least 2 times a whole working day (~10 hours) ; coming back home gparted was still "waiting" for the end of the parted process. I honestly don't think it would have ever finished...
Created attachment 154939 [details] cout debug log After adding some cout lines: it's indeed hanging in cleanup_cursor in Utils.cc in the second for loop. This might be caused by the string being worked on having a length of over 12 millions (only 5,000 when cloning an empty partition). While this isn't noticeable when printing to stdout, logging the output results in a 11MB log file. Debug output attached.
Thank you nicobo, Hernando, and James for staying active with this bug report. And an huge thank you to Hernando for your work to pin-point where the GParted code is "stuck". The cleanup_cursor method tries to remove the extraneous percent complete information from the output so that it will be short enough to fit in the gparted_details.htm log file. I have been working on a way to spawn the command asynchronously so that I can provide better progress information. My thought was to add this progress information on the screen somewhere, perhaps in a separate window. If / when I get this working, I could clean up the output on the fly. This would prevent having to sort through a 12 million byte string and hopefully avoid this problem. On a different note, what commands or steps did you use Hernando to capture the back traces?
I simply attached gdb to the gpartedbin process once the ntfsclone process was gone (gdb --pid=<PID of gpartedbin>). I then used Ctrl+c and 'c' to interrupt and continue the process and 'thread apply all bt (full)' to get the backtraces.
Thanks Hernando for the gdb tips. :) I have reworked the cleanup_cursor logic to minimize the number of string erase operations performed. This updated code is available at: http://gparted.sourceforge.net/curtis/gparted-0.5.1-20100301.tar.bz2 Would you be able to download, compile, and test this code with your system to see if it addresses the problem?
Yupp, it works! [DBG] Entering cleanup_cursor [DBG] String length before first for loop: 12428851 [DBG] String length before second for loop: 12428851 [DBG] String length after second for loop: 326 [DBG] String value: NTFS volume version: 3.1 Cluster size : 4096 bytes Current volume size: 209909141504 bytes (209910 MB) Current device size: 209909145600 bytes (209910 MB) Scanning volume ... 100.00 percent completed Accounting clusters ... Space in use : 203595 MB (97.0%) Cloning NTFS ... 100.00 percent completed Syncing ... [DBG] Leaving cleanup_cursor Took only a couple of seconds. :-)
Thanks for the quick confirmation. :) Since the problem is not related to 1 TB disk drives, but rather large NTFS partitions (e.g., > 100 GB), I will edit the original title to reflect this discovery.
This enhancement has been committed to the GNOME git repository for inclusion in the next release of GParted (0.5.2). The relevant git commit can be viewed at the following link: http://git.gnome.org/browse/gparted/commit/?id=260e0ea1902e066bfe233e4a505ae800755572bc Closing this bug.