After an evaluation, GNOME has moved from Bugzilla to GitLab. Learn more about GitLab.
No new issues can be reported in GNOME Bugzilla anymore.
To report an issue in a GNOME project, go to GNOME GitLab.
Do not go to GNOME Gitlab for: Bluefish, Doxygen, GnuCash, GStreamer, java-gnome, LDTP, NetworkManager, Tomboy.
Bug 575324 - Copying a huge NTFS partition (e.g., > 100 GB) never finishes
Copying a huge NTFS partition (e.g., > 100 GB) never finishes
Status: RESOLVED FIXED
Product: gparted
Classification: Other
Component: application
0.4.3
Other All
: Normal normal
: ---
Assigned To: gparted maintainers alias
gparted maintainers alias
: 611284 (view as bug list)
Depends on:
Blocks:
 
 
Reported: 2009-03-14 04:01 UTC by James N. Walters
Modified: 2010-03-01 21:02 UTC
See Also:
GNOME target: ---
GNOME version: ---


Attachments
Detail log from successfully coping an NTFS partition (8.13 KB, text/html)
2009-03-15 14:52 UTC, Curtis Gedak
Details
gparted waiting at the end of the operation (21.99 KB, image/png)
2010-01-19 21:57 UTC, nicobo
Details
destination disk shows the new partition in the background (30.52 KB, image/png)
2010-01-19 21:58 UTC, nicobo
Details
output of ps -ef after operation cancelled and gparted closed (14.85 KB, text/plain)
2010-01-19 22:37 UTC, nicobo
Details
before starting the cloning (15.09 KB, text/plain)
2010-01-21 07:22 UTC, nicobo
Details
during the cloning (with disk activity) (15.29 KB, text/plain)
2010-01-21 07:23 UTC, nicobo
Details
after the cloning (gparted still waiting) (15.40 KB, text/plain)
2010-01-21 07:24 UTC, nicobo
Details
after the cloning (gparted closed) (14.61 KB, text/plain)
2010-01-21 07:25 UTC, nicobo
Details
gparted log (3.29 KB, text/html)
2010-01-21 07:26 UTC, nicobo
Details
backtrace while hanging (2.04 KB, text/plain)
2010-02-28 19:32 UTC, Hernando Torque
Details
full backtrace, note frame 2 of thread 2 (43.59 KB, text/plain)
2010-02-28 20:05 UTC, Hernando Torque
Details
cout debug log (1.25 KB, text/plain)
2010-03-01 10:58 UTC, Hernando Torque
Details

Description James N. Walters 2009-03-14 04:01:56 UTC
Please describe the problem:
When using the UI to copy the partition, it runs for a long time and then stops all disk activity, ntfsclone is nowhere evident in the task list or top but the UI never moves on to the next step in the task.



Steps to reproduce:
Try to copy a huge NTFS partition (>750 GB) to another disk > 1TB using an MS-Dos partition table.
 


Actual results:
It never finishes after the ntfsclone step; the UI refuses to continue (no system issues, no crash, no UI freeze - it can be exited and all the menus work).

Expected results:
The partition is copied correctly and the application returns to ready state.

Does this happen every time?
Yes

Other information:
This is an MS-DOS Partition table. I am running 32bit XP and cannot use GPT, even if I wanted to, which I shouldn't have to. This can't be the 2TB limit because I am only copying 750 GB and the target disk is 1.5TB only.

The only way I got this critical task of mine to work (I NEEDED to copy this partition!) was to look at the code, and then copy a much smaller partition which did finish. Then I was able to duplicate the commands on the command line and I was able to finish it.  Then, going back into the UI to grow the partition to the full 1.5 TB worked.

This was the same whether it was with the latest version of the LiveCD (0.4.3-2) or Parted Magic running the latest version of gParted (0.4.3). I even tried an earlier version of the LiveCD (0.4.1-2) but same results.

I saw somewhere in the forum that someone had reported this but didn't create a bug and the developer didn't acknowledge this.

Rest assured, this is a real bug. A blocker for NTFS users with huge disks.
Comment 1 Curtis Gedak 2009-03-14 14:45:02 UTC
Thank you James for reporting this bug and providing a detailed description.

After you could not find ntfsclone running and shut down gparted, do you recall if there was a copy of the partition on the new 1.5 TB disk?

What I am trying to discover is if ntfsclone completed successfully or not when run from within the gparted application.

Could you provide the output from running the following command in a terminal window?

     fdisk -l -u
Comment 2 James N. Walters 2009-03-14 20:25:37 UTC
Yes, sorry I forgot that.

After all the times that this happened, there was, in fact a copy of the partition on the new disk.

However, since I was not sure of the state of that partition and did not want to spend another number of hours doing a compare, I can't be sure if the partition was viable or not. It seemed to be but I didn't want to risk it.

Like I said before, after a few hours when it looked to have completed, ntfsclone was not evident in the 'ps agx' list or 'top', so it looked like it completed successfully (?), but gparted did not see it for some strange reason and could not continue with its validation of the new partition and update of the partition table as normal.

As for the 'fdisk -l -u', be advised that I needed to have this partition working, so I did this manually, as I said before, and finished the process via the command line (including writing to the partition table with dd as gparted would have done if this bug was not there.)

So, I now have a working disk and partition and am running XP Pro, no problem.  Would the output of 'fdisk -l -u' really be of any use to you from this?  I can do that if you want.  This involves booting up from the gparted disk again and running it.

What I _can't_ do is attempt the partition copy again just so I can give you the output since I have been running on it a few days and have new data that I don't want to lose.

This looks like a strange one, because I could not tell from the code that anything would cause gparted to hang on the ntfsclone invocation unless ntfsclone is not returning the correct value upon completion or never really completes but hangs for some reason but hidden from the process table?
Comment 3 Curtis Gedak 2009-03-15 14:52:04 UTC
Created attachment 130699 [details]
Detail log from successfully coping an NTFS partition

Attached for reference is a gparted details log from successfully copying a partition.

When gparted invokes external commands such as ntfsclone, it spawns a shell that executes the command. When the command finishes, it passes an exit code to the shell.  Then the shell finishes and passes the exit code to gparted.  Hence if GParted was reporting that it was running the ntfsclone command, and ntfsclone did not show up in the process list, then why would the shell not have finished and returned an exit code to gparted?  This is strange.

Unfortunately I do not enough drive space to test copying large NTFS partitions of the size you indicated.  ;-(

To figure out what went wrong is difficult without being able to reproduce the problem.

The reason I asked for an "fdisk -l -u" listing is that this would show the exact sector boundaries of the partitions.  This may or may not be relevant, but is additional information that could be collected and might help lead to a solution.
Comment 4 nicobo 2010-01-19 21:54:08 UTC
Hello

I can confirm this bug. It is happening to me with the copy of a 230G ntfs partition.

I can reproduce the problem 1 or 2 times for you but please ask me quickly as I'll soon need to use the disk (and will therefore use another way).

Here are first some infos I could gather while the operation was finished but gparted still waiting :

The destination disk is a fresh new 500G disk on which I first made backups of another computer (the last 2 partitions : sdc1 and sdc2).
I've created them at the beginning of the disk then I moved them to the end to give room for the next one at the beginning of the disk.

The operation "simply" consists in copying a whole disk (sdb1, its only 230G partition) to the beginning of the new one (the 500G one).

Both disks are plugged in my computer via USB 2.

I'm using gparted 0.4.5-2ubuntu1 (karmic).

I've previously made the same operation with smaller partitions (10G and 100G as seen in the following outputs) and it worked really fine, so the problem *seems* to live between 100G-230G for me.

Some screenshots :
http://img248.imageshack.us/img248/6534/screenshotapplyingpendi.png (main, background window)
http://img163.imageshack.us/img163/9922/destinationdisk.png (destination disk, as the filename indicates)

No ntfsclone in the process list.

But gparted is still using 85% of the CPU (result of the "top" command) :

PID USER PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
26650 root 20 0 74488 47m  13m S 87.5  2.4 538:02.61 gpartedbin                 
1247 root 20 0 55668  18m 9.9m R  7.3  0.9 236:35.88 Xorg
...

Here is the output of "fdisk -l -u" for the concerned disks :

root@fondcombe:~# fdisk -l -u

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x000628d9

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63   488392064   244196001    7  HPFS/NTFS

Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x44fdfe06

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1       737801190   758284064    10241437+   7  HPFS/NTFS
/dev/sdc2       758284065   976768064   109242000    7  HPFS/NTFS
/dev/sdc3              63   488392064   244196001    7  HPFS/NTFS

Partition table entries are not in disk order
Comment 5 nicobo 2010-01-19 21:57:30 UTC
Created attachment 151791 [details]
gparted waiting at the end of the operation
Comment 6 nicobo 2010-01-19 21:58:09 UTC
Created attachment 151792 [details]
destination disk shows the new partition in the background
Comment 7 Curtis Gedak 2010-01-19 22:05:43 UTC
Thank you nicobo for adding to this bug report.  Following are some requests:

1)  Would you be able to provide a complete process listing after this problem has occurred?

For example:

    $ ps -ef

I am curious as to what other processes may still be outstanding, even in the ntfsclone process has disappeared.


2)  Does the copy of the NTFS file system seem to be OK?

Can you mount the file system, view your files, etc?


3)  Would you be able to try the ntfsclone from the command line?

For example using the command from your attached picture:

    $ ntfsclone -f --overwrite /dev/sdc3 /dev/sdb1

After the command completes also execute:

    $ echo $?

This will display the return code from the previous command.
Comment 8 nicobo 2010-01-19 22:36:00 UTC
1) see new attachment (was done in the same X session, but after having closed gparted ; I can re-do it after the next try if needed)

2) Yes, the copy *seems* ok and I can mount the partition and access my files. By the way there are way too many files for me to check if everything is ok by hand and I don't know of a quick way to do it...

3) I come back to you asap
Comment 9 nicobo 2010-01-19 22:37:13 UTC
Created attachment 151794 [details]
output of ps -ef after operation cancelled and gparted closed
Comment 10 Curtis Gedak 2010-01-19 22:39:49 UTC
1)  If possible would you be able to retry the copy?  Then while GParted is still up and running could you snag a complete process listing "ps -ef"?

I am curious to see if there is a GParted spawned shell process still outstanding.
Comment 11 nicobo 2010-01-20 07:21:01 UTC
3) partition sdc3 deleted via gparted ; new empty partition created via fdisk (new part. starting at 1 and with exact same size as the source, then type set to 7 HPFS/NTFS) ; then :

root@fondcombe:~# fdisk -l -u

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x000628d9

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63   488392064   244196001    7  HPFS/NTFS

Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x44fdfe06

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1       737801190   758284064    10241437+   7  HPFS/NTFS
/dev/sdc2       758284065   976768064   109242000    7  HPFS/NTFS
/dev/sdc3              63   488392064   244196001    7  HPFS/NTFS

Partition table entries are not in disk order
root@fondcombe:~# ntfsclone -f --overwrite /dev/sdc3 /dev/sdb1
ntfsclone v2.0.0 (libntfs 10:0:0)
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 250056699904 bytes (250057 MB)
Current device size: 250056705024 bytes (250057 MB)
Scanning volume ...
100.00 percent completed
Accounting clusters ...
Space in use       : 247608 MB (99.0%)   
Cloning NTFS ...
100.00 percent completed
Syncing ...
root@fondcombe:~# echo $?
0
root@fondcombe:~# fdisk -l -u

Disk /dev/sdb: 250.1 GB, 250059350016 bytes
255 heads, 63 sectors/track, 30401 cylinders, total 488397168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x000628d9

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63   488392064   244196001    7  HPFS/NTFS

Disk /dev/sdc: 500.1 GB, 500107862016 bytes
255 heads, 63 sectors/track, 60801 cylinders, total 976773168 sectors
Units = sectors of 1 * 512 = 512 bytes
Disk identifier: 0x44fdfe06

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1       737801190   758284064    10241437+   7  HPFS/NTFS
/dev/sdc2       758284065   976768064   109242000    7  HPFS/NTFS
/dev/sdc3              63   488392064   244196001    7  HPFS/NTFS

Partition table entries are not in disk order
Comment 12 Curtis Gedak 2010-01-20 17:16:44 UTC
From comment #11 I see that the ntfsclone command did complete successfully.  Since GParted spawns a shell to execute commands, I wonder if the shell is still lingering around when this problem occurs?

It would be appreciated if you could retry the copy using GParted and provide the output from a "ps -ef" before closing GParted.
Comment 13 nicobo 2010-01-21 07:22:59 UTC
Created attachment 151922 [details]
before starting the cloning
Comment 14 nicobo 2010-01-21 07:23:41 UTC
Created attachment 151923 [details]
during the cloning (with disk activity)
Comment 15 nicobo 2010-01-21 07:24:44 UTC
Created attachment 151924 [details]
after the cloning (gparted still waiting)
Comment 16 nicobo 2010-01-21 07:25:17 UTC
Created attachment 151925 [details]
after the cloning (gparted closed)
Comment 17 nicobo 2010-01-21 07:26:15 UTC
Created attachment 151926 [details]
gparted log
Comment 18 nicobo 2010-01-21 07:27:30 UTC
Hi

1) I've attached the "ps -ef" results as 4 files : before, during, waiting and after the operation ; plus the detailed gparted log of the whole operation

Hope this helps
Comment 19 Curtis Gedak 2010-01-23 00:20:40 UTC
Thank you nicobo for providing all of these various process listings.

Both the ntfsclone process, and the shell from which it was spawned have disappeared from the process list.  Meanwhile gparted is still waiting for the spawned shell to close.  This is very strange behaviour indeed.

At the moment I cannot think of any other requests to ask of you to narrow down this bug.  I realize that you most likely need to get on with using your system so please feel free to do so.

Thanks again for adding more information to this bug.  I will have to ponder this a while to think of what the next step might be.
Comment 20 Hernando Torque 2010-02-27 20:19:33 UTC
Any news on this?

I can reproduce this bug by copying a 120GB NTFS-partition from a 640GB disk to a 1TB disk (copying a 50GB partition works fine). Once the I/O-activity stops, the RAM usages goes down, CPU usage increases, and according to the system monitor gparted is in a sleeping state (waiting channel: hrtimer_nanosleep). I used ntfscmp to check the copy and it seems complete.

I'm using gparted 0.5.1 with Ubuntu 10.04 Alpha 3.
Comment 21 Curtis Gedak 2010-02-28 00:18:10 UTC
So far I have been unable to replicate the problem.

I have two 160 GB drives.  Today I tried creating a 120 GB NTFS partition on one, and then using GParted to copy from one to the other.  The operation went smoothly and was successful.

How much of your NTFS partition is in use?
Comment 22 Hernando Torque 2010-02-28 16:22:53 UTC
The partition is pretty much full (120/126GB used), as I resized it before backing it up to the 640GB disk.

Maybe worth mentioning: gpartedbin keeps hogging CPU even after cancelling the supposedly unfinished operation.
Comment 23 Curtis Gedak 2010-02-28 16:52:49 UTC
*** Bug 611284 has been marked as a duplicate of this bug. ***
Comment 24 Hernando Torque 2010-02-28 17:32:32 UTC
FWIW, copying an empty 126GB partition works, while copying the same partition with 10GB data (created some files using dd if=/dev/zero) triggers the bug.
Comment 25 James N. Walters 2010-02-28 17:47:14 UTC
Heh, it took almost a year, but I am now finally getting confirmation of this bug from others. Oh well. I guess being an early adopter of technology (huge - >1TB disks) sometimes doesn't pay.

As I said before in my previous expositions about this bug, it looks like parted itself is not returning a proper status or perhaps any status at all when it exits, causing the outer shell of gparted to continue waiting forever?

Can this bug at least be now marked as "confirmed"?  >1TB disks are now pretty cheap, so testing by the developer should be able to be done now.
Comment 26 Hernando Torque 2010-02-28 18:17:44 UTC
Sorry for the misinformation: copying the partly filled 126GB partition to the 1TB disk did work - it just took more time than I thought (while showing the same "resource" symptoms as the bug).

This brings me to the question, if the operation really wouldn't finish or if it would just take abnormal long? I let it run no longer than an hour once the ntfsclone process was gone before cancelling the operation. Think I'll let it run longer now.
Comment 27 James N. Walters 2010-02-28 18:24:21 UTC
I let it run for quite a few hours, a few times, without it finishing.
Comment 28 Hernando Torque 2010-02-28 19:32:00 UTC
Attached gdb to gpartedbin: it seems to hang in the cleanup_cursor method of Utils.cc. The value of 'str' for IA__g_utf8_pointer_to_offset changes constantly (the percentage value of the string jumps around - it doesn't increase or decrease).
Comment 29 Hernando Torque 2010-02-28 19:32:58 UTC
Created attachment 154907 [details]
backtrace while hanging
Comment 30 Hernando Torque 2010-02-28 20:05:53 UTC
Created attachment 154909 [details]
full backtrace, note frame 2 of thread 2
Comment 31 nicobo 2010-03-01 02:22:14 UTC
FYI, I ran the test at least 2 times a whole working day (~10 hours) ; coming back home gparted was still "waiting" for the end of the parted process.

I honestly don't think it would have ever finished...
Comment 32 Hernando Torque 2010-03-01 10:58:03 UTC
Created attachment 154939 [details]
cout debug log

After adding some cout lines: it's indeed hanging in cleanup_cursor in Utils.cc in the second for loop. This might be caused by the string being worked on having a length of over 12 millions (only 5,000 when cloning an empty partition). While this isn't noticeable when printing to stdout, logging the output results in a 11MB log file.

Debug output attached.
Comment 33 Curtis Gedak 2010-03-01 15:50:54 UTC
Thank you nicobo, Hernando, and James for staying active with this bug report.  And an huge thank you to Hernando for your work to pin-point where the GParted code is "stuck".

The cleanup_cursor method tries to remove the extraneous percent complete information from the output so that it will be short enough to fit in the gparted_details.htm log file.

I have been working on a way to spawn the command asynchronously so that I can provide better progress information.  My thought was to add this progress information on the screen somewhere, perhaps in a separate window.  If / when I get this working, I could clean up the output on the fly.  This would prevent having to sort through a 12 million byte string and hopefully avoid this problem.

On a different note, what commands or steps did you use Hernando to capture the back traces?
Comment 34 Hernando Torque 2010-03-01 16:15:25 UTC
I simply attached gdb to the gpartedbin process once the ntfsclone process was gone (gdb --pid=<PID of gpartedbin>). I then used Ctrl+c and 'c' to interrupt and continue the process and 'thread apply all bt (full)' to get the backtraces.
Comment 35 Curtis Gedak 2010-03-01 18:50:43 UTC
Thanks Hernando for the gdb tips.  :)

I have reworked the cleanup_cursor logic to minimize the number of string erase operations performed.  This updated code is available at:

http://gparted.sourceforge.net/curtis/gparted-0.5.1-20100301.tar.bz2

Would you be able to download, compile, and test this code with your system to see if it addresses the problem?
Comment 36 Hernando Torque 2010-03-01 20:22:03 UTC
Yupp, it works!

[DBG] Entering cleanup_cursor
[DBG] String length before first for loop:
12428851
[DBG] String length before second for loop:
12428851
[DBG] String length after second for loop:
326
[DBG] String value:
NTFS volume version: 3.1
Cluster size       : 4096 bytes
Current volume size: 209909141504 bytes (209910 MB)
Current device size: 209909145600 bytes (209910 MB)
Scanning volume ...
100.00 percent completed
Accounting clusters ...
Space in use       : 203595 MB (97.0%)   
Cloning NTFS ...
100.00 percent completed
Syncing ...

[DBG] Leaving cleanup_cursor

Took only a couple of seconds. :-)
Comment 37 Curtis Gedak 2010-03-01 20:34:30 UTC
Thanks for the quick confirmation. :)

Since the problem is not related to 1 TB disk drives, but rather large NTFS partitions (e.g., > 100 GB), I will edit the original title to reflect this discovery.
Comment 38 Curtis Gedak 2010-03-01 21:02:23 UTC
This enhancement has been committed to the GNOME git repository for inclusion in the next release of GParted (0.5.2).

The relevant git commit can be viewed at the following link:
http://git.gnome.org/browse/gparted/commit/?id=260e0ea1902e066bfe233e4a505ae800755572bc

Closing this bug.