Syncing Scrivener on iOS takes a long time

[Katherina_74]

I have not logged in here in awhile, and I now see that other people have the same problem, I decided to slice up my one big file. So I essentially made many copies of it and in each one I deleted all but Project A in one of the copies, Project B in the other, etc. The problem, however, persists for me. Alas, as is, Scrivener on my iphone is not usable. I am yet to experiment to see if I remove all files and create one brand new small one if the syncing problems will persist.

I use Devon Think Pro Office, and initially it was such a headache to sync the files from my laptop to my iphone, but now they created an option with a Bonjour connection which works fine. I did have to get a faster internet connection, and the syncing lasts for awhile, but it is convenient because when I am out, I do not have to worry about getting interrupted with syncing,. A Bonjour connection for Scrivener would work for me (if that ever becomes an option) because it is also more secure. I can sync before I leave home, and sync back again when I come back home.

[rdale]

I too have been experiencing slower syncs (taking over a 1 minute while "Downloading file list" before it starts syncing just a few changes). I also just had an issue where a file it was trying to sync was missing. Since I hadn't done any work on my iphone lately, I just unlinked dropbox from iOS Scrivener, deleting the local copies, and then re-linked. The long sync times continued, but I had no expectation that they would be faster--the main problem of not being able to sync at all was fixed though. Also, all my project names were lower-case, instead of their original title case. I checked on Dropbox.com, and that's how they showed up there (on my Mac, they were correctly title-cased), but upon syncing with the iOS version of Scrivener, the lower case project names showed up. I had to remove the projects from Dropbox and re-add them to get the title case restored.

But upon seeing shalnya's suggestion, I fired up a utility I wrote to look for dropbox "conflicted" files, which have occasionally tripped me up, but had not in a long time. Sure enough, a conflicted "recents.txt" file had showed up sometime in late December. Deleting that file seems to have solved the issue for me. Now my syncs with existing, unmodified projects take about 15 Seconds.

For comparison with your setups; I have 3 projects currently synced up.
Their sizes are 1.2M, 14M, and 61M, with 208, 503, and 2086 files & folders, respectively.

I wonder if whatever happened to make Dropbox change my scrivener project filenames (folders, actually) lower-case has to do with this issue? Is it related to my conflicted file?

As for the rest of you, have you been able to search through all of your project's inner structure for files with 'conflicted' in their names? If not, here's a command line that will locate them for you:

Open Terminal, and type the two-letter command 'cd', and then hit the space bar.
In the Finder, locate your Dropbox Scrivener sync folder, and drag it into the Terminal application. You should see the path appear after the 'cd' command--something looking like this:

Code: Select all

Simon:~ rguthrie$ cd /Users/rguthrie/Dropbox/Apps/Scrivener


Hit enter. Then paste the following non-destructive command into the terminal:

Code: Select all

find . -iname "*conflicted*"


If it shows you any paths and file names, then you have conflicted files and need to get rid of them. Hopefully that helps some of you at least. If it outputs nothing, then you have no files or folders with the word "conflicted" in their names.

While it's not true for me, I wonder if the issue arose for some of you because you converted projects from version 2 to version 3 in an actively syncing Dropbox folder? Seems like the process of copying the original project and then converting it, as new files and folders are being moved, created, and modified, is a situation rife with the potential for sync problems.

[shenzo]

I have just tried shalnya's suggestion with the help of rdale's command line tips. I was hopeful it would help me use Scrivener 3.0 + iOS. Unfortunately, it didn't work for me. Here's my experiment: I have 15 projects of various sizes (from 6MB to 1GB), all upgraded to Scrivener 3.0.1. Before upgrading them, I duplicated them in a different folder to be still formatted for Scrivener 2.9. If I change a single file in one project, the "Downloading files list" phase takes about 15min for the 3.0 formatted projects. The 2.9 formatted projects only take 25sec. In both cases, the projects are not opened on Scrivener for Mac. I can't see any correlation with project file size and number of files. It seems more related to the way 3.0 synchronizes projects.

[naquada]

I'm following this as well.. as syncing on iOS seems to take an age... the downloading file list takes some time (few minutes) and then the syncing files takes a while (but not as long as the downloading file list)...

I've moved everything out of drop box and moved it back in... still the same issue...

I've run the command line to check for the *conflict* files.. and there are lots.. but loading into scrivener doesnt show up and conflict boxes... can we just delete the *conflict* files?!!?

something is very screwy.. and its not as easy to just fire up iOS scrivener and make some notes, when you're waiting a good 5-10 mins for the thing to sync ;(

[rdale]

I've run the command line to check for the *conflict* files.. and there are lots.. but loading into scrivener doesnt show up and conflict boxes... can we just delete the *conflict* files?!!?

... Probably? If Scrivener isn't recovering conflicted files for you to examine, then the files that are conflicted are probably infrastructure files that Scrivener isn't using (such as the conflicted version of the recents.txt file that I deleted). What I'd do is locate your automatic backups folder (see Scrivener->Preferences->Backups for a button to open that location in the finder). Then use File->Back Up->Back Up To... to create a backup, adding '-with-conflicted-files' as part of the name, just in case something goes horribly wrong.

Then close scrivener, sync any last changes you have on iOS, unlink iOS Scrivener from your iOS device (deleting local copies when prompted), and then go about deleting those conflicted files on your Mac. Keep re-running the command line "find" command using the up-arrow button to bring it back up, until you've tracked down and deleted every conflicted file.

When you're done with each project,load it into Mac Scrivener and check the contents of as many files as you can, just in case. After you're done with all that... I'd check dropbox.com to see if you had a similar problem that I did (with lower-casing of your projects). If all seems well there, re-link dropbox on your iOS device, and let your projects re-sync.

I hope that helps. Good luck!

[shenzo]

Someone from the staff in this thread reported a while ago that something about this issue is being worked out 'behind the scenes', could we have an update on a possible resolution of these delays? Would love to use Scrivener 3 in the way I use Scrivener 2 now, i.e. lighting fast syncing! Thanks.

[GrahamGrün]

Just adding that I as well experience the annoying „downloading files list“ waiting time and I as well hope for the team to find out what the problem is.

[sidderke]

Just to add some additional feedback to the conversation:

I also tried the whole deleting some recent.conflict files in the Scrivener packages but it didn't help.

I also have the feeling that the syncing, next to the fact that it used to take some seconds on whichever hardware, now it takes about 3 minutes on an iPhone 8 Plus, and about 10-15 minutes on an iPad Air. I'm talking only about the 'downloading file list' thing.

[kewms]

I've run the command line to check for the *conflict* files.. and there are lots.. but loading into scrivener doesnt show up and conflict boxes... can we just delete the *conflict* files?!!?

If you have a lot of conflict files, that will definitely affect sync performance, Scrivener has to check to make sure they aren't "real." See rdale's post for the best approach to clearing them out.

Katherine

[AmberV]

Another command-line that might be useful, simply to get an idea of how much content is being worked with:

Code: Select all

find ~/Dropbox/Apps/Scrivener | wc -l

The first part of that is the path to the Dropbox folder, and it should be adjusted if you aren’t using defaults. This command finds all files and directories within that folder and generates an output that prints each file name, one per line. The second part of the command is a simple counting utility, set to count the total lines of output, thus a total count of files and folders. That’s how much it has to process when downloading a file list and going through each item, checking modification dates and so forth to see what needs to be downloaded and what can be ignored.

Scrivener projects can be a lot bigger than you might think in terms of total internal files and folders. For example, if I run that command on the stock interactive tutorial all by itself, I find it has 278 files and folders—and I’d bet that’s a pretty typical project size in terms of quantity. If you have even half a dozen projects around that size, you’re looking at close to 2,000 files and folders. If I were to drop this one single project that I’m typing into to the sync folder, it would add 9,110 files to be looked at, every single time.

Now to get a sense of the difference between Scrivener 2 and 3, if I export this project to 2.0 format and check its total file & folder count, I’ll come up with 5,056—a little over half the size in sheer quantity. So for purely processing the bulk of the project every time I sync, the v3 format version of it requires about twice as much effort.

That’s obviously a somewhat extreme example, but the more projects you have, and the more binder items you have in each project, the greater the effect on sync times.

Things that can add a lot of content to your folder:

• Snapshots. And if you have the option set to snapshot all documents changed when syncing, every sync adds more snapshots. Use the new Snapshots Manager to clean out old snapshots you no longer need.
• Backups. I don’t just mean the automatic backups (which shouldn’t be anywhere near this folder), I mean if you have a v2 project and you load it in v3, the first thing it does is make a duplicate copy. Leave that backup sitting around in your sync folder and that’s an extra chunk of files to be processed that probably serve no purpose being there.
• Conflict files. As noted that’s a good thing to check for anyway. Scrivener 3 should be much better at detecting these and helping you clean them up, but until you do you’ll have more content than you strictly need.

As to why Scrivener needs to download all of this:

There are two layers going on here. With the project list, we have a defined list of folders stored in the device that we can poll Dropbox with independent queries for. So when the app reaches a condition where it checks, it just goes down that list and checks the modification for the container project folder itself (or it might check the .scrivx, I’m not sure on that one, either way it doesn’t have much to check) against what is stored on the phone, and draws an icon, throws a warning, etc. For a small list of items like that, this kind of check is okay.

It’s however a very expensive check for bulk stuff. For each folder you have to send an individual request to Dropbox for information on a single file. So that’s where we get the second layer—once you punch the sync button we now are faced with a few problems:

1. We don’t know what files are on Dropbox. We could make some pretty good assumptions with the project list that there are folders on Dropbox with names matching the projects, but as to all of the pieces inside the folders? Who knows—and that’s not counting new projects that may not be in the list yet. We can’t send out a request for a modification date on a folder unless we know the folder or file exists.
2. The method Dropbox provides to use for gathering information about the sync folder is all or one-thing. We use the one-thing method to quickly check status, but that method is wholly inappropriate for downloading what may potentially be thousands of pieces of metadata. It is by far more efficient to ask Dropbox for a full manifest of every file in the sync folder.

Moral of the story: keep your sync folder tidy! It was designed to be used for only active working projects you want to sync. Everything else in that folder that you never open on iOS, that is just sitting there for half a year, that is an archived version, etc., is deadweight.

[Hugh]

Thanks for the explanation, Ioa.

[sidderke]

Amber, thanks for the explanation. But I do think still something is up. I have been told before that the Scrivener 3 format doubles the amount of files that need to be checked.
But the time difference in the 'downloading file list' between Scrivener 2 and 3, if you look at the test users did, was, in percentage, a difference of 1200 percent sometimes. Much more then the 200 expected with a doubling of files to be checked.

I still hold hopes that the development team can find why the difference is so big in sync time.

[GrahamGrün]

If it’s not a bug but an inherent problem of the way Scrivener syncs, I really hope you can find a way around it soon. I know and appreciate that stability has top priority for you. But a mobile version is not of much use if it is impossible to spontaneously access your project. And people get tempted to just skip the sync, add something to the project and hope for Scriveners great capability for resolving conflicts later.

[lunk]

... or maybe the solution is not to have several hundreds of projects or super big projects with lots of research material in them, if you want to sync with Dropbox fairly quick. That seems to me a simpler solution and you can apply it today.

[AmberV]

It would help us to have some concrete description of what it takes to make things slow down so much; what thresholds we are talking about. I’m currently running a test on another issue, and it involves around 500 internal files in total. For me syncing is nearly instantaneous. I provided some tools earlier in the thread for monitoring a full file count, but I’ve yet to see any results posted from that.