.scrivx file truncated

User avatar
Three Blind Lice
Posts: 73
Joined: Thu Apr 02, 2009 4:25 pm
Platform: Windows

Sat Oct 28, 2017 12:35 am Post

Hi,

Something spooky just happened to me - my main project file just wouldn't open (or more precisely, opening it would crash Scrivener). I poked arount a bit and finally discovered that the .scrivx file was truncated. The last line just ended like this:

Code: Select all

    <PrintSettings PaperType="A4" LeftMargin="0" RightMargin="0" TopMargin="0" BottomMargi


I checked out a working .scrivx file and completed the final line to read like this:

Code: Select all

    <PrintSettings PaperType="A4" LeftMargin="72.0001" RightMargin="72.0001" TopMargin="90.0002" BottomMargin="0" Orientation="Portrait" Collates="No" ReverseOrder="No"/>
</ScrivenerProject>


Seems like I got away with this - everything working just fine again. Still, I find this rather scary. How can this happen in the first place? Isn't there supposed to be some kind of security system in place (e.g. saving data to a separate file first and then renaming it)?

User avatar
kewms
Posts: 3228
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sat Oct 28, 2017 3:44 pm Post

Are you sharing the project via Dropbox or a similar service? Incomplete synchronization would be one potential cause of this.

Katherine
Scrivener Support Team

User avatar
Three Blind Lice
Posts: 73
Joined: Thu Apr 02, 2009 4:25 pm
Platform: Windows

Sat Oct 28, 2017 4:06 pm Post

No, actually I don't. I have my projects on an SD card that I swap between different computers. Typically, I shut down the computer properly before swapping the SD, but more often than not, Scrivener still keeps complaining that the project is already open on a different machine. Up until last week, this has never caused any problems, however.

Either way, I would find it very important if incomplete synchronization (or whatever else) couldn't lead to potential corruption and/or loss of an entire project, even if there is an unforeseen problem. To me, this looks like the procedure needs to be redesigned in a way that this is impossible.

User avatar
xiamenese
Posts: 3403
Joined: Mon Jan 29, 2007 1:32 am
Platform: Mac
Location: London or Exeter, UK.

Sat Oct 28, 2017 4:32 pm Post

Scrivener complaining that it’s open on another machine means that at some point Scrivener hadn’t completed its cleaning up as it was saving on closing down and the user.lockfile that it creates when you open a project was not deleted as should happen. SD cards are generally slow, so it seems to me, you’ve been being a tad too quick in shutting down your computer and removing the SD card before Scrivener has fully completed its background processes.

Do tell me I’m wrong.

Mark
The Scrivenerato sometimes known as Mr X.
MacBook Pro Retina 10.13, 8GB RAM, 512GB SSID
MacBook Pro 17" (late 2011) 10.13, 8GB RAM, 512GB SSID
iPad Air 2, iOS 11, 64GB
Scrivener, Scapple, Nisus Writer Pro, Bookends …

User avatar
Three Blind Lice
Posts: 73
Joined: Thu Apr 02, 2009 4:25 pm
Platform: Windows

Sat Oct 28, 2017 5:05 pm Post

Well probably - only that it's definitely not me who is too quick, but either Windows not waiting for the process to shut down. Or maybe the fact that the Scrivener file is in an encrypted container. Or possibly something else.

All kinds of possible reasons, really. But again, this is not about picking one, but about the fact that Scrivener should be robust enough to preserve the project in any of these scenarios. I mean, here's another possible problem: power outage during "cleanup". Do you really want to rewrite your entire novel just because some idiot in your apartment block flipped the switch at the wrong moment? Imho it's really a matter of program design to keep your work out of harms way.

User avatar
lunk
Posts: 1740
Joined: Wed Aug 21, 2013 4:24 pm
Platform: Mac + iOS
Location: Sweden 64° N

Sat Oct 28, 2017 6:00 pm Post

In the unlikely event that the power outage happens exactly when you are closing the project you do have backups, don't you? With normal HDDs och SSDs it is so little that is written to disk that it takes very little time.

When you say you shut down the computer, does it mean that you actually finish Windows and turn the PC off?

User avatar
Three Blind Lice
Posts: 73
Joined: Thu Apr 02, 2009 4:25 pm
Platform: Windows

Sat Oct 28, 2017 6:20 pm Post

lunk wrote:When you say you shut down the computer, does it mean that you actually finish Windows and turn the PC off?

No of course not, I typically just take a pair of scissors and cut the cable.

lunk wrote:In the unlikely event that the power outage happens exactly when you are closing the project you do have backups, don't you? With normal HDDs och SSDs it is so little that is written to disk that it takes very little time.

You mean backups that already exist before the original file was even written properly? You could earn a lot of money with something like that, you know.

I would appreciate if this thread could stay on the constructive side. I asked politely, support responded politely. I think we were making good progress. There is no need for anyone to intervene just because they don't happen to have the exact same problem that I have.

User avatar
lunk
Posts: 1740
Joined: Wed Aug 21, 2013 4:24 pm
Platform: Mac + iOS
Location: Sweden 64° N

Sat Oct 28, 2017 6:51 pm Post

Some say that they shut Windows off when they in fact only close the lid on a laptop. People sometimes don’t do exactly what they say, or simplify their descriptions off what happened in a way that makes it harder to understand exactly what happened.

Backups are by default created every time you close the project and you would have closed the project before closing Scrivener and before turning Windows off, right?

...and, in a user forum you should expect users to respond.

User avatar
Three Blind Lice
Posts: 73
Joined: Thu Apr 02, 2009 4:25 pm
Platform: Windows

Sat Oct 28, 2017 7:49 pm Post

lunk wrote:you would have closed the project before closing Scrivener and before turning Windows off, right?

Says who?

lunk wrote:Some say that they shut Windows off when they in fact only close the lid on a laptop. People sometimes don’t do exactly what they say, or simplify their descriptions off what happened in a way that makes it harder to understand exactly what happened.

...and, in a user forum you should expect users to respond.

Right. But are you aware that you are actually sabotaging my thread? The issue I posted was about Scrivener not being as robust as it could be. To avoid misunderstandings, I even repeated that in my second post. And I posted this in "Bug Hunt" and not in "Technical Support". For a reason. Yet you insist on turning this thread into something completely different. I understand the reasoning behind your questions - but you don't seem to understand that your questions are off topic.

Crash handling and recovery is really crucial in word processing programs, and among other things, it's also a matter of respect towards a progam's users (=yes, your work is important to us). "You should have backups, ya know!" just isn't a valid answer to this. If it's user generated backups - well, they aren't really helpful if the files are corrupt and the program doesn't even tell you, because eventually the good copies will be overwritten with the corrupt ones, and that's that. If its automatic backups by Scrivener - at least in my case, the latest zips were definitely older than the corrupted version. But why should I have to face data loss in the first place? Look at Word, Openoffice Writer or Libreoffice - you never lose anything. If you start up the program and there was any kind of problem, you will be immediately notified AND presented with repair options (not that I've ever seen the original file become corrupt - but maybe I've simply been luckier there).

Bottom line is that my .scrivx became corrupt and I would have lost part of my work if I hadn't been able to manually edit the underlying file. To me, that's just not how it should be.

User avatar
devinganger
Posts: 903
Joined: Sat Nov 06, 2010 1:55 pm
Platform: Mac + Windows
Location: Monroe, WA USA
Contact:

Sat Oct 28, 2017 8:42 pm Post

Three Blind Lice wrote:But why should I have to face data loss in the first place? Look at Word, Openoffice Writer or Libreoffice - you never lose anything. If you start up the program and there was any kind of problem, you will be immediately notified AND presented with repair options (not that I've ever seen the original file become corrupt - but maybe I've simply been luckier there).

Bottom line is that my .scrivx became corrupt and I would have lost part of my work if I hadn't been able to manually edit the underlying file. To me, that's just not how it should be.


HAHAHAHAHAHA.

Word and Open/Libre Office most certainly DO lose/corrupt files, and their automatic repair options are not even close to being "mostly" successful. If I could get back the time I've spent reconstructing missing data from those programs, I'd be years younger.

You have just run into two of the design philosophies behind Scrivener that make it fairly unique:

1) Use well-known file formats, not proprietary formats, so that if something goes wrong users have options to recover their data.
2) Break the data up into lots of smaller files, so that if something goes wrong, the impact is confined to a handful of files.

KB could probably spend the time to write a wiz-bang recovery wizard, but that's time he wouldn't be able to write any of the rest of Scrivener goodness. The nice thing is, he doesn't have to (as you have just demonstrated) -- something went wrong and because of how a Scrivener project is constructed, you were able to find your data.

As to having to face data loss in the first place -- that is part and parcel of using a computer. Even the big word processors can't protect against some of the most common data loss scenarios (they try by doing tricks like taking backup copies of your file behind your back), and some of those scenarios are because of user error while others are because of timing issues that are very trcky to work out even with decades of experience and code addressed at solving them.

To minimize the chances of data loss, follow best practices like "make sure your application has completely written files to your SD card before sleeping/powering down the system."
--
Devin L. Ganger
Devin on Earth: http://www.devinonearth.com/
Plotter on the streets, pantser in the sheets

User avatar
kewms
Posts: 3228
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sat Oct 28, 2017 9:16 pm Post

Three Blind Lice wrote:Well probably - only that it's definitely not me who is too quick, but either Windows not waiting for the process to shut down. Or maybe the fact that the Scrivener file is in an encrypted container. Or possibly something else.

All kinds of possible reasons, really. But again, this is not about picking one, but about the fact that Scrivener should be robust enough to preserve the project in any of these scenarios. I mean, here's another possible problem: power outage during "cleanup". Do you really want to rewrite your entire novel just because some idiot in your apartment block flipped the switch at the wrong moment? Imho it's really a matter of program design to keep your work out of harms way.


Scrivener's project folder is not encrypted.

Plenty of people using Word have lost important data because "some idiot flipped the switch at the wrong moment." That's why uninterruptible power supplies and backup disks exist. A device that requires electricity to function will cease functioning if electricity stops flowing. A data transfer that requires network access will fail if the network access is interrupted. These are fundamental limitations that apply in one way or another to all software and all computing devices.

Katherine
Scrivener Support Team

User avatar
kewms
Posts: 3228
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sat Oct 28, 2017 9:26 pm Post

Three Blind Lice wrote:Right. But are you aware that you are actually sabotaging my thread? The issue I posted was about Scrivener not being as robust as it could be. To avoid misunderstandings, I even repeated that in my second post. And I posted this in "Bug Hunt" and not in "Technical Support". For a reason. Yet you insist on turning this thread into something completely different. I understand the reasoning behind your questions - but you don't seem to understand that your questions are off topic.


It is not off topic to attempt to establish whether the observed behavior is, in fact, a bug.

Crash handling and recovery is really crucial in word processing programs, and among other things, it's also a matter of respect towards a progam's users (=yes, your work is important to us). "You should have backups, ya know!" just isn't a valid answer to this. If it's user generated backups - well, they aren't really helpful if the files are corrupt and the program doesn't even tell you, because eventually the good copies will be overwritten with the corrupt ones, and that's that. If its automatic backups by Scrivener - at least in my case, the latest zips were definitely older than the corrupted version. But why should I have to face data loss in the first place? Look at Word, Openoffice Writer or Libreoffice - you never lose anything. If you start up the program and there was any kind of problem, you will be immediately notified AND presented with repair options (not that I've ever seen the original file become corrupt - but maybe I've simply been luckier there).


Having personally lost a very substantial amount of work in Word, I can only envy your good fortune.

Bottom line is that my .scrivx became corrupt and I would have lost part of my work if I hadn't been able to manually edit the underlying file. To me, that's just not how it should be.


If you feel this conversation has become unproductive, you should feel free to contact the support team directly.
http://www.literatureandlatte.com/suppo ... ction-bugs

I think we can all agree that a corrupt .scrivx file is bad. But it's somewhat difficult to troubleshoot without a clear and reproducible description of the circumstances that caused the corruption to occur. Identifying those circumstances may, in turn, illustrate ways in which the user can minimize the risk to their data while a possible fix is being investigated.

Katherine
Scrivener Support Team

User avatar
lunk
Posts: 1740
Joined: Wed Aug 21, 2013 4:24 pm
Platform: Mac + iOS
Location: Sweden 64° N

Sat Oct 28, 2017 9:37 pm Post

Three Blind Lice wrote:
lunk wrote:you would have closed the project before closing Scrivener and before turning Windows off, right?

Says who?
.


Scrivener is designed to work on one machine but projects can safely be shared using e.g. Dropbox or removable media, provided that the project is proparly closed before another instance of Scrivener accesses the project. Getting the message that the project is still open elswhere suggest that this is not the case. Scrivener is very robust but like any software it is possible to make it fail.

User avatar
Three Blind Lice
Posts: 73
Joined: Thu Apr 02, 2009 4:25 pm
Platform: Windows

Sat Oct 28, 2017 10:54 pm Post

kewms wrote:Scrivener's project folder is not encrypted.

On my SD card, it is. :wink:

kewms wrote:Plenty of people using Word have lost important data because "some idiot flipped the switch at the wrong moment." That's why uninterruptible power supplies and backup disks exist. A device that requires electricity to function will cease functioning if electricity stops flowing. A data transfer that requires network access will fail if the network access is interrupted. These are fundamental limitations that apply in one way or another to all software and all computing devices.

Yes and no. To my knowledge, the way it works with the programs mentioned is that the data is written to a temporary file first, then the files are renamed and the old file is deleted. That way, there's always a complete file, regardless what electricity does (that is, unless electricity fries the disk). I was under the impression that this is good practice, but it appears that Scrivener doesn't do that, otherwise I don't see how a file could be truncated in this way.

But yeah, I take back what I said about data not being lost in the programs mentioned. It's just that I use Scrivener for most of my writing now, so I probably forgot about anything frustrating that happened in the past. And I totally agree about the benefits of splitting everything into smaller chunks, but that doesn't mean there isn't room to improve. Also granularity is part of the reason why it would be a major headache to lose the structure that ties it all together (with some stuff, losing the structure would be almost as bad as losing the texts themselves).

kewms wrote:It is not off topic to attempt to establish whether the observed behavior is, in fact, a bug.

Unless someone questions the corruption itself, this is a matter of definition, not analysis. Maybe I wasn't clear enough in the beginning, but to me, the fact that this particular type of corruption (incomplete file and constant crashing that can only be fixed with a file editor) occurred points to a design flaw, regardless of what led to the corruption. However, if the programmers say that is how it is supposed to be, then it's not a bug, regardless of what I think of it.

kewms wrote:But it's somewhat difficult to troubleshoot without a clear and reproducible description of the circumstances that caused the corruption to occur. Identifying those circumstances may, in turn, illustrate ways in which the user can minimize the risk to their data while a possible fix is being investigated.

Ironically, that is part of the design flaw to me, because since Scrivener didn't alert me to the corruption and I worked on other stuff in the meantime, I didn't notice until 1-2 weeks later. After that time, of course, it is more or less impossible to troubleshoot the circumstances.

lunk wrote:Scrivener is designed to work on one machine but projects can safely be shared using e.g. Dropbox or removable media, provided that the project is proparly closed before another instance of Scrivener accesses the project. Getting the message that the project is still open elswhere suggest that this is not the case. Scrivener is very robust but like any software it is possible to make it fail.

True. But it seemed like you were suggesting that it is necessary to manually close documents before closing Scrivener, and to close Scrivener before shutting down Windows. Is that indeed good practice? It seems to me that Scrivener will take the time to do Backups and shut down properly even if you shut down Windows directly, on the other hand I do know that Windows 10 will sometimes not wait for applications to close properly (used to happen with Firefox) so these programs will behave like they crashed when you open them next time. Unfortunately, that's where my knowledge ends - no clue if that only happened in the early stages of Windows 10, and whether or not it's something to expect from Windows, or if it can be avoided on the application end.

User avatar
kewms
Posts: 3228
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sun Oct 29, 2017 12:01 am Post

Three Blind Lice wrote:Yes and no. To my knowledge, the way it works with the programs mentioned is that the data is written to a temporary file first, then the files are renamed and the old file is deleted. That way, there's always a complete file, regardless what electricity does (that is, unless electricity fries the disk). I was under the impression that this is good practice, but it appears that Scrivener doesn't do that, otherwise I don't see how a file could be truncated in this way.


And if the power drops while the temporary file is being written?

Generally speaking, any program will load the "working" file into the computer's memory, make changes there, and then write the edited file to disk. If the power drops while there are changes in memory that have not yet been written to disk, those changes will be lost.

A Scrivener project does in fact contain multiple copies of the .scrivx file. Without more information, it's impossible to say exactly what led to the damage you observed, or to Scrivener's inability to recover from it.

Unless someone questions the corruption itself, this is a matter of definition, not analysis. Maybe I wasn't clear enough in the beginning, but to me, the fact that this particular type of corruption (incomplete file and constant crashing that can only be fixed with a file editor) occurred points to a design flaw, regardless of what led to the corruption. However, if the programmers say that is how it is supposed to be, then it's not a bug, regardless of what I think of it.


But *what* design flaw? For you as the user, it may be enough to say, "the file was corrupted, therefore Scrivener messed up." But in order to actually address the problem, it's necessary for the development team to identify a specific situation in which a particular action did not occur as intended, and therefore a specific chunk of source code that, say, failed to detect and alert the user to an error.

Consider the case of a novel reader who says, "I just didn't find the protagonist believable." Now, as the author, fix the character. Good luck.

Ironically, that is part of the design flaw to me, because since Scrivener didn't alert me to the corruption and I worked on other stuff in the meantime, I didn't notice until 1-2 weeks later. After that time, of course, it is more or less impossible to troubleshoot the circumstances.


Did you work on other things in the same project? If so, then the corruption probably had not yet occurred.

True. But it seemed like you were suggesting that it is necessary to manually close documents before closing Scrivener, and to close Scrivener before shutting down Windows. Is that indeed good practice? It seems to me that Scrivener will take the time to do Backups and shut down properly even if you shut down Windows directly, on the other hand I do know that Windows 10 will sometimes not wait for applications to close properly (used to happen with Firefox) so these programs will behave like they crashed when you open them next time. Unfortunately, that's where my knowledge ends - no clue if that only happened in the early stages of Windows 10, and whether or not it's something to expect from Windows, or if it can be avoided on the application end.


If Windows 10 is known to not allow programs to close properly, then it would be a good idea to manually close one's Windows 10 applications. Laptops and removable disks also present special problems, as the user can physically interrupt what might otherwise be an orderly shutdown process by closing the lid or physically disconnecting the disk.

Regardless of whether it's necessary to manually close programs in the abstract, doing so can also be a useful troubleshooting tool. If the problem never occurs when you close Scrivener manually, but happens one in every three or four times when you just close Windows, then that's a pretty clear indication of where the problem might lie.

Katherine
Scrivener Support Team