tomkarpik.com » Massive Data Loss Bug in Leopard

Massive Data Loss Bug in Leopard

November 5th, 2007

Update: The bug occurs regardless of the type of destination being moved to (whether it’s local USB, local Firewire, SMB, etc.). Also, I have been informed that this bug goes back all the way to Panther.

Update 2: Here’s a video of the same thing happening to a USB drive: leopardmovebug.avi

Update 3: Robert Rodgers adds: “I accidently chmod’d a directory while a massive (40+GB) file move was being copied into it to be -w by me, files went ‘poof’.”

Leopard’s Finder has a glaring bug in its directory-moving code, leading to horrendous data loss if a destination volume disappears while a move operation is in action. I first came across it when Samba crashed while I was moving a directory from my desktop over to a Samba mount on my FreeBSD server.

I’ve now run tests on a Windows XP SP2 SMB mount, as well as a local HFS+ formatted USB drive, and the bug surfaces every time the destination disappears while the Finder is moving something to the destination.

Details

Before I go into more detail, let’s clear up what it means to “copy” or “move” a file:

Copying from/to the same volume: A new file reference is created on the file system, and then data is read from the source and written to the destination.

Moving from/to the same volume: The file reference is modified to reflect the new path. No data is ever read/written.

Copying from/to different volumes: A new file reference is created on the file system of the other volume, and then data is read from the source on the original volume, and written to the destination.

Moving from/to different volumes: A new file reference is created on the file system of the other volume, and then data is read from the source on the original volume, and written to the destination. If the previous operation is successful, the original file on the original volume is deleted.

As you can see, moving a file from one location to another on the same volume is a piece of cake. Moving a file between two volumes really incorporates two file system operations: a copy, and then a delete … but only if the copy was successful (at least on a sane system!).

In terms of pseudo code, moving from/to different volumes might look something like this:

move(source, destination) {
    returnVal = copy(source, destination);

    if(returnVal == TRUE) {      // copy success
        delete(source);
    } else {                     // copy fail
        message("Move failed");
    }
}

This is where the problem with the Mac OS X 10.5.0 Finder lies.

Demonstration

The following is a simple demonstration that anyone with a SMB server (be it Windows itself, Samba on FreeBSD/Linux, etc.) can duplicate. The specifications for the demonstration were as follows:

Client

MacBook Pro (2.0 GHz Core Duo, 2 GB of RAM)
Mac OS X Leopard (10.5.0)

Server

Same MacBook Pro running VMware
Windows XP SP2

For my purposes, I simply booted up Windows XP in VMware and connected to it as if it were a whole separate computer (OS X doesn’t know the difference).

Let’s go through the regular rigmarole of finding a Windows PC’s IP and connecting to it through SMB (“Windows File Sharing”).

Great. We now have C:\ mounted on our Mac running Leopard. Let’s prepare for the simulated SMB mount failure by opening Computer Management\System Tools\Shared Folders\Sessions in Windows.

Our Mac’s connection to the share should be obvious.

Let’s now open a Terminal on our Mac, and create a directory with some garbage data in it. Generate enough data so that a move operation will take at least 10 seconds. In my case, my test file is 384 MB.

(The command I used to generate the test directory with a garbage file is as follows: mkdir test && dd if=/dev/random of=test/testfile bs=4096 count=98304)

You’ll now see a "test" directory on your Desktop.

Here’s the key step. Move the "test" directory from your Desktop to the C$ mount by holding Command (Apple) and dragging. Dragging without holding Command will initiate a copy operation (that’s what the plus means), so be sure to hold Command until you release the drag.

You’ll know Finder expects to perform a move when there is no plus beside the directory you’re dragging.

Okay, our move operation is doing its thing. (Notice how the title of the window still says “Copy”? Remember what I said about moving files across volumes?)

Quickly go back to the Windows session, right-click on the active share session, and close it.

Finder will error out with one of its typically useful error messages and codes. This much is to be expected. The "test" directory can still be seen on our Desktop.

Aaaand … poof!

The directory on our Desktop is now gone. The directory shown in the C$ mount will only contain files that were copied entirely before the connection was disrupted, but nothing more (in the demonstration case, there’s only one file in "test", so nothing was copied successfully).

Where’s the rest of our data?

Apple, please FTFF already. This is unacceptable.

Words From Others

Massive Data Loss Bug in Leopard - MacNN Forums

November 5th, 2007

[…] wrote a fairly detailed article about it on my site (here: tomkarpik.com � Massive Data Loss Bug in Leopard), but the gist of it is basically … 1. Initiate move operation across volumes 2. Destroy the […]

Stephen Chiang

November 5th, 2007

Yeah, I found this bug the hard way on Friday when I was moving a couple of large MP4 files to an external eSATA drive and the copy failed during the move (a different unrelated bug that occurs when a large file is transferred to a Seagate 750GB Freeagent Pro drive using the eSATA interface when connected to a Tempo SATA Express34 card in Leopard). I reported both these incidents via ADC bug report website on Friday evening. I hope these bugs get fixed soon. Coming from a Windows environment, I actually use move quite a bit!

Andrei Kolev

November 5th, 2007

Thanks for the tip! I was considering myself a power user and I had access to prerelease versions but I was not aware that there is such a feature in Leopard. Well, I am not going to use it yet but I hope it will be fixed soon.

David Hopper

November 5th, 2007

Well this would absolutely explain why my /Library/Receipts directory was empty after my Archive and Install. The Installer crashed at the end of the process, deleting my Receipts from both source and destination.

The problem is that this is baked into the Installer DVD. They’ll have to recall / cut new ones once this bug is eliminated.

Bob Wallace

November 5th, 2007

I encountered this problem last December while setting up a new iMac running Tiger. Never imagined it would be so common or repeatable. Thanks for documenting a repeatable scenario.

Jean-Francois Roy

November 5th, 2007

Please file a Radar at http://bugreport.apple.com. It’s the only way this is going to be fixed promptly.

Zero Gravitas

November 5th, 2007

I hate to say but this bug has been around since Panther.

Steven Fisher

November 5th, 2007

This was in Tiger, too. Does Apple actually document that you can hold down the command key to move the file instead of copying? (Not that I think it shouldn’t be fixed, but I’m a little less concerned if a feature that doesn’t officially exist doesn’t work well.)

Scott Perry

November 5th, 2007

Welcome to 2003.

This is not a Leopard regression and thus is not a good reason not to upgrade.

Timmy

November 5th, 2007

Did you check the trash?

Bill McCloskey

November 5th, 2007

So, if I do this from the command prompt i.e. # mv test /Volumes/smbfs, is there an error?

John

November 5th, 2007

Just curious, as I’m too lazy to set up my own test environment…does the same problem occur when you’re in a terminal session and use ‘mv’ to move the files?

Just curious.

Tom

November 5th, 2007

Aha. I wasn’t aware that this bug existed pre-Leopard.

It’s still something that should have been fixed, given the fact that the Leopard Finder has received the biggest overhaul in Mac OS X history.

whatithink » Blog Archive » Leopard: Data Loss Bug

November 5th, 2007

[…] With any new OS there’s bound to be a few problems. Now that hype is dying down a bit, and as Leopard user get down to using the new OS, the bugs are beginning to surface. The biggest one reported to date involves data loss when moving files between different file systems. […]

codepoetry | I Don’t Like Leopard

November 5th, 2007

[…] what else is stupid? Apparently there’s a massive data loss bug in the new Finder. For years — literally, we’ve been complaining about how awful the […]

lesterbangs

November 5th, 2007

this is completely off-topic but how did you get the neato colorized prompt in yer bash shell? i know you do it via a ‘export prompt’ command but i’m too lazy to figure out the magic incantation.

DTNick

November 5th, 2007

Does this issue only affect SMB file sharing? And if you list the ~/Desktop/ directory in the Terminal, is the test folder still missing??

tsuehpsyde.com » Blog Archive » Mac OS X Data Loss Bug

November 5th, 2007

[…] is far from flawless. First, we had OS X upgrades giving users the infamous BSOD, and now we have a data loss bug as well. The mockery of the Windows BSODby Apple in 10.5 seems all too ironic if you ask […]

dtNick

November 5th, 2007

^^ignore my first question. I see it was answered at the very beginning of the article.

Zaphod

November 5th, 2007

You’re unacceptable. -Steve J.

Leopard 網絡抄檔嚴重漏洞 at MacGrass

November 5th, 2007

[…] 有人發現在 Leopard ，使用拖拉方法將檔案由本地主機抄到網絡上的 SAMBA 分享處／另一個宗卷（Volume），事後本地資料會被刪除掉。（是刪除，不是在垃圾箱）就算網絡抄錄不成功，例如抄寫期間 SAMBA 失效、網絡故障，網絡機器固然不能獲得本地主機的資料，本地資料一樣會被刪除。試想像你想將你的 iPhoto library 抄到網絡上的 NAS ，抄了兩個檔案後 NAS 失效所帶來的沉重代價。本網忠告市民，在同類型的網絡備份作業，可先將要備份的檔案打包（如用 Zip 、 tar 之類），再傳送打包檔案。這樣可減低資料缺失的可能性。 […]

Mike

November 5th, 2007

Could this be why I lost 100 Itunes songs and my Entourage archive file of all my emails?

Tom

November 5th, 2007

lesterbangs: Drop me an e-mail and I’d be glad to help you out.

dtnick: It appears to affect any destination type — so pulling the plug on a USB drive would cause it to happen as well.

Dave Winer isn’t happy with Leopard. Is that Apple’s fault, or is it bigger than that? « GracefulFlavor

November 5th, 2007

[…] Vista sure wasn’t. And now, the more you look, the more you see that Leopard is getting its share of bitching and grumbling. […]

Ric Ford

November 5th, 2007

See a very similar data-loss bug from Mac OS X 10.1 here:

http://www.macintouch.com/Archives/ReaderReports/MacOSX/early/mosxreader10.1.3pt08.html

Snak

November 5th, 2007

Good reason to stay away from anything Windows.

Hardware 2.0 mobile edition

November 5th, 2007

[…] issue has been covered in detail over on Tom Karpik’s blog. Here’s the lowdown on this bug: Leopard’s Finder has a glaring bug in its […]

dallas

November 5th, 2007

I know this bug and it definitely is not new. It seems to me it was Tiger, though I cannot rule out that it was Panther. Anywa, I lost a huge number of pictures from a flash drive a while back because I was doing a move and bumped the ccable and the drive lost connection to the computer. Really stinks to put it mildly. I never use move based on that experience. I use only copy and then go back and delete the files at a later time.

Z

November 5th, 2007

What wallpaper are you using on the virtual pc?

Jason’s Blog! » Massive Data Loss Bug in Leopard

November 5th, 2007

[…] read more | digg story […]

Tom

November 5th, 2007

z: That would be http://psychopulse.deviantart.com/art/The-Bait-N-C-Edition-48178244

JasonF

November 5th, 2007

Wow, you got /.’d tom. Nice one.

Does Leopard really suck? : The Last Podcast

November 5th, 2007

[…] Every day now, I hear how badly Apple’s Leopard sucks. Now there is news of a massive data loss bug. […]

Zimmie

November 5th, 2007

I wouldn’t call this a massive data loss bug. You have to go out of your way to move a file to a remote share rather than copy it. My pet ACL bug is comparable.

mkdir test cd test touch Foo chmod +a “[yourUserNameHere] deny delete” Foo rm Foo

The system should return “rm: Foo: Permission denied”.

cd .. rm -rf test

The system should return “rm: test/Foo: Permission denied”, then “rm: test: Directory not empty”.

Now, drag the directory ‘test’ to the Trash and empty it. The directory and the file in it will be deleted.

In both cases, you have to be doing something weird to encounter the bug.

danoz

November 5th, 2007

I ran into this myself just the other day, never had this problem on any previous version of os x. I was moving about 2Gb of files from my desktop to an smb share, it failed with an i/o error of some sort and poof they all disappeared, very frustrating seeing as i was putting them on the share so that they got properly backed up.

danoz

November 5th, 2007

zimme: how is holding down the apple key to perform a move instead of a copy going out of my way? i’ve done this thousands of times in the past and never had this problem before…

Stan’s List » Blog Archive » Verified data loss bug in Leopard’s Finder

November 5th, 2007

[…] Tom Karpik came across the problem when his Samba crashed (file and print services). OS X | Oodles | trackback No Responses to ‘Verified data loss bug in Leopard’s Finder’ […]

W. Ian Blanton

November 5th, 2007

This was apparently reported on MacInTouch on Nov. 2nd. I’ve confirmed it by cutting SMB services while transferring, and it definitely happens in Leopard. I’ve also confirmed it does not affect 10.4(.9) Client.

Additionally, if you MOVE several folders, and the connection is cut, only one folder is affected. This is of course, still bad, but not as bad as it could be. Apple definitely needs to fix this.

Steve Cummings

November 5th, 2007

Hmm,

For the present I find myself with all Tiger home network, in the past I could have tried to repeat this on a BSD, Linix, Windows setup.

Has anyone tried the disappearing volume move on another BSD platform at least? Windows.

It would be nice to move the data first, and then reclaim space once completed (with checksum?).

But, I’m going to ask those with multiple platform systems what their experiences are.

Solaris anyone?

Jonathan

November 5th, 2007

Have you submitted this to Apple via bugreport.apple.com? They’re usually fairly responsive to formally submitted bugs.

Hamranhansenhansen

November 5th, 2007

The bug to me is that there is a move command at all between volumes. You can’t move stuff between volumes, only copy it.

What is the advantage of the AVI movie over ISO MPEG-4 H.264/AAC?

MacVillage.de » Schwerer Bug im Leopard-Finder

November 5th, 2007

[…] vom 6 November 2007 07:25 Schwerer Bug im Leopard-Finder Laut tomkarpik.com beinhaltet der Leopard-Finder einen schwerwiegenden Fehler, der im schlimmsten Fall zu einem […]

David Newall

November 5th, 2007

I think what is occurring is an effect of write-back caching, which operating systems use to improve performance.

When you ask to write a file, the operating system (OS) stores the change in RAM, but delays actually writing to disk until later. The OS reports a successful write and lets you continue with your work, on the assumption that the disk-updates really will take place as expected. This enormously improves performance. It lets you continue your work which, in the case of a move across volumes, would be to remove the original file (this also will be reported as successful before actually being committed to disk.)

Without realising it, you caused the loss of data by interrupting access to the disk (or server.) You can probably specify a mount property or flag, probably called “synchronous”, which will physically write to disk (or server) before permitting you to continue your work; it really will slow you down.

ty

November 5th, 2007

fwiw: I just tested this on Tiger 10.4.10 (MBP CD2) by connecting to an AFP volume (on a different Tiger box) and turning off airport in the middle. Got the Finder error dialog, but it didn’t delete the test folder.

[图]Mac OS X Leopard 爆缺陷可能导致数据丢失_WuGuiSuDu.com

November 5th, 2007

[…] 技术人员发现苹果最新操作系统Mac OS X Leopard 的 Finder 文件夹移动代码中有一个明显的缺陷,这可能会导致大批数据丢失. 作者在文中截图展示了Mac OS X Leopard在把文件夹通过Samba复制到挂载在另一台FreeBSD服务器的目录时侯出现问题,数据竟然意外丢失了. 内含原文链接. 查看:Massive Data Loss Bug in Leopard […]

Massive Data Loss Bug in Leopard

Details

Demonstration

Client

Server

Words From Others

Andrei Kolev

David Hopper

Bob Wallace

Zero Gravitas

Scott Perry

Timmy

Zaphod

Mike

Snak

dallas

Z

Zimmie

danoz

danoz

Jonathan

Hamranhansenhansen

ty

Zimmie

michael

M!KE

sip

robert

Erik

Mark Petereit

encosion

Nick

mike

copyCat

Oolie

copyCat

Will

zahadum

rei

Kenny

Kenny

Mr Big

Simon

Loweded Wookie

Loweded Wookie

Loweded Wookie