Syncing Scrivener on iOS takes a long time

User avatar
kewms
Posts: 4124
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Thu Dec 14, 2017 4:59 pm Post

Katherina_74 wrote:How can I resolve a conflict in Scrivener? I only have two files in the Conflicts folder.


Are you getting the Resolve Conflicts window in desktop Scrivener? Did you follow the directions?

What should I do with the Recovered folders files? I work with links a lot and I do not want to delete these files because I am afraid it will mess up the links.


Well, if these files are outside the Binder structure, the links from the main project are already messed up. I would recommend (with Scrivener closed) dragging these folders out to the main desktop and inspecting the contents to see what's there.

p.s. I have to admit I am a bit disappointed that 5000+ files is too many for Scrivener. I am hoping that in another upgrade this won't be a problem, especially now that different projects can be viewed with tabs.


Sorry, but there is no foreseeable upgrade path that will make a project with 5000 files sync as fast as one with 50 files. Scrivener has nothing to do with it, really, it's just the nature of how file transfers work.

Katherine
Scrivener Support Team

mi
mijalce
Posts: 16
Joined: Fri Aug 14, 2015 9:55 pm
Platform: Mac

Fri Dec 15, 2017 10:44 am Post

I am facing the same problem.
I do agree that it is not the same to sync 5000 with 50 files, however I had the same number of files with Scrivener 2. Then there were no delays in sync with iOS or 3 minutes Downloading file list on iOS. Thus, I deduce that there is some change in Scrivener 3 that triggered this. Additionally, I noticed the following when I open the Scrivener 3 project on macOS it changes immediately
- user.lock
- recents.txt
- binder.backup
- myProject.scrivx
These files are uploaded on Dropbox. I make no changes to any file or settings in the Scrivener 3 project and then I close it. When I close it on Dropbox are uploaded the following things
- docs.checksum
- search.indexes
- myProject.scrivx
- Preview.html
- Thumbnail.jpg
- ui-common.xml
- binder.autosave
- ui.plist
Then I go to my iOS Scrivener and shows me that there is a change on the macOS and I need to sync, although I made no changes at all. I start the sync and it lasts for 3 minutes. Majority of time (approximately 99% of time) is the status "Downloading file list". The actual sync of the "changed" files is 5 seconds.
Hope this makes contribution in tackling this issue. Thanks!

is
istara
Posts: 65
Joined: Sat Nov 04, 2006 7:07 pm

Fri Dec 15, 2017 11:25 am Post

I have the same problem.

It clearly started a week ago, with the update.

Scrivener is likely checking every single file in every single project every time, not just the one(s) it has identified as actually needing updating. That is why the problem may only be affecting those of us with a large number of large Project files.

This did NOT happen previously. It has managed to sync quickly for an entire year, and now it's taking over two minutes in my case just to check the files list (after which it takes about 5 seconds to actually sync).

Whatever you did to fix the DropBox sync bug has possibly done this. I really hope you can fix it urgently, because waiting two minutes for a sync is frustratingly long when you want to get going quickly in a work meeting etc.

NB: the Project file I have been testing this on has zero conflicts. So that is not the issue. Also, Project files in the past that did have conflicts did not experience this problem.

User avatar
kewms
Posts: 4124
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Fri Dec 15, 2017 4:59 pm Post

Scrivener has to check the entire file list. Scrivener on the iOS device has no way of knowing what might have changed on the Dropbox server unless it checks.

Scrivener 3's file format is a bit more complex, in part to make it more robust against synchronization errors.

If you are seeing constant reports of changes on the iOS device even if there were no changes elsewhere, it wouldn't hurt to clear the Dropbox cache. You can do that via the 'Reset Scrivener' command in the Scrivener section of the iOS Settings app.

Katherine
Scrivener Support Team

is
istara
Posts: 65
Joined: Sat Nov 04, 2006 7:07 pm

Sat Dec 16, 2017 12:02 am Post

kewms wrote:Scrivener has to check the entire file list. Scrivener on the iOS device has no way of knowing what might have changed on the Dropbox server unless it checks.
Katherine


But it has presumably checked the file list already, and identified which project(s) need updating. Why, when it has done that, is it apparently rechecking the entire folder of projects rather than just the single project file it needs to update?

I can't see why a process that used to take seconds is now taking over two minutes. Something was actually UN-fixed in the last update. Software shouldn't be getting slower in subsequent versions.

User avatar
kewms
Posts: 4124
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sat Dec 16, 2017 12:04 am Post

It should only be checking the one project file, but it will check all items within that project. That's the behavior I'm seeing on my system. See my previous note on resetting the sync cache if you are having issues.

Katherine
Scrivener Support Team

User avatar
lunk
Posts: 2831
Joined: Wed Aug 21, 2013 4:24 pm
Platform: Mac + iOS
Location: Sweden 64° N

Sat Dec 16, 2017 12:08 am Post

istara wrote:
Why, when it has done that, is it apparently rechecking the entire folder of projects rather than just the single project file it needs to update?

The project isn’t a single file. Each entry in the binder within one project is a file.
I am a user, writing non-fiction and science, using:
* Mac Scrivener 3 on a Macbook 12”, MacBook Pro 13”, and iMac 27”, all running the latest MacOS
* iOS Scrivener 1 on an iPhone 8, iPad Air 9.7”, and iPad Pro 12.9”, all running the latest iOS

is
istara
Posts: 65
Joined: Sat Nov 04, 2006 7:07 pm

Sat Dec 16, 2017 3:21 am Post

kewms wrote:It should only be checking the one project file, but it will check all items within that project. That's the behavior I'm seeing on my system. See my previous note on resetting the sync cache if you are having issues.

Katherine


I tried this. There is no improvement.

lunk wrote:
istara wrote:
Why, when it has done that, is it apparently rechecking the entire folder of projects rather than just the single project file it needs to update?

The project isn’t a single file. Each entry in the binder within one project is a file.


I'm well aware of this. But it did NOT previously take two minutes to do this check. Why has this problem only started since the previous update?

I suspect it is actually checking every file in EVERY project, not just the single project it is supposed to be updating.

To test this further:

1. I created a brand new project file - TEST.scriv - which has a minimal amount of sub files due to being brand new and essentially empty. It is 98kb in size and I have written one word in it.

2. I synced the iPad so it appears there.

3. I reopened it on Desktop and added one more word. Closed it. Watched it sync back up to DropBox (which appeared to include updating about 11 files, but took only a couple of seconds).

4. I opened up Scrivener iOS. It takes about a second to check the files, and identify that just one of them - TEST.scriv - needs updating.

5. I pressed to sync, and it takes 2 minutes 20 seconds to do "Checking file list" and then ~5 seconds to update TEST.scriv

Can you not see from this behaviour that Scrivener is clearly checking every single file in every single project instead of just the Project that it has already identified needs updating?

User avatar
kewms
Posts: 4124
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sat Dec 16, 2017 5:26 am Post

istara wrote:4. I opened up Scrivener iOS. It takes about a second to check the files, and identify that just one of them - TEST.scriv - needs updating.

5. I pressed to sync, and it takes 2 minutes 20 seconds to do "Checking file list" and then ~5 seconds to update TEST.scriv

Can you not see from this behaviour that Scrivener is clearly checking every single file in every single project instead of just the Project that it has already identified needs updating?


No, I can clearly see that something is causing synchronization to take an unusual amount of time on your system.

I just did a similar test. I opened my two main projects on the desktop. They're both almost entirely text, with a combined file size of about 3 MB. I added one file and one line to each. I closed them and allowed Dropbox to sync.

Then I opened Scrivener on the iPad and allowed it to sync. In addition to these two, it has 3 other projects. All five combined total about 39 MB. 15 seconds start to finish. (I should note here that I have a very fast internet connection.)

So yes, the behavior that you are seeing is clearly anomalous. But might I respectfully suggest that arguing about how Scrivener does or does not work behind the scenes is probably not conducive to getting it solved?

There are a few possibilities:

* For whatever reason, Scrivener's indicators for what files need to be synchronized are inaccurate, causing unnecessary and time consuming checks. Clearing the Dropbox cache should have resolved this.

* For whatever reason, there are excess files either inside the individual Scrivener projects or in the Dropbox/Apps/Scrivener folder as a whole. I would recommend inspecting the Apps/Scrivener folder both on the Mac and via the Dropbox web interface to see if there are any files that look like they shouldn't be there. I would also recommend checking the overall project sizes in Finder to make sure they seem reasonable.

* If neither of those helps, a more drastic solution would be as follows:

-- Ensure that all projects are synchronized to the Mac. Move them *out* of the Dropbox folder to another location.
-- Synchronize the now empty Dropbox folder with the iPad. Ensure that it is still empty after you're done, that nothing has re-appeared from the iPad. If anything has, either delete it or move it to a non-Dropbox location.
-- Re-install iOS Scrivener on the iPad. Configure synchronization, but there shouldn't be anything there.
--- One by one, move your projects back to Dropbox, Sync to the iPad -- which *will* take some time, since it's the whole project -- then make a minor change and sync again. The second sync should be very fast. If it isn't, you've identified which project is the source of the problem.

Katherine
Scrivener Support Team

is
istara
Posts: 65
Joined: Sat Nov 04, 2006 7:07 pm

Sat Dec 16, 2017 11:20 am Post

kewms wrote:
istara wrote:4. I opened up Scrivener iOS. It takes about a second to check the files, and identify that just one of them - TEST.scriv - needs updating.

5. I pressed to sync, and it takes 2 minutes 20 seconds to do "Checking file list" and then ~5 seconds to update TEST.scriv

Can you not see from this behaviour that Scrivener is clearly checking every single file in every single project instead of just the Project that it has already identified needs updating?


No, I can clearly see that something is causing synchronization to take an unusual amount of time on your system.

I just did a similar test. I opened my two main projects on the desktop. They're both almost entirely text, with a combined file size of about 3 MB. I added one file and one line to each. I closed them and allowed Dropbox to sync.

Then I opened Scrivener on the iPad and allowed it to sync. In addition to these two, it has 3 other projects. All five combined total about 39 MB. 15 seconds start to finish. (I should note here that I have a very fast internet connection.)

So yes, the behavior that you are seeing is clearly anomalous. But might I respectfully suggest that arguing about how Scrivener does or does not work behind the scenes is probably not conducive to getting it solved?

There are a few possibilities:

* For whatever reason, Scrivener's indicators for what files need to be synchronized are inaccurate, causing unnecessary and time consuming checks. Clearing the Dropbox cache should have resolved this.

* For whatever reason, there are excess files either inside the individual Scrivener projects or in the Dropbox/Apps/Scrivener folder as a whole. I would recommend inspecting the Apps/Scrivener folder both on the Mac and via the Dropbox web interface to see if there are any files that look like they shouldn't be there. I would also recommend checking the overall project sizes in Finder to make sure they seem reasonable.

* If neither of those helps, a more drastic solution would be as follows:

-- Ensure that all projects are synchronized to the Mac. Move them *out* of the Dropbox folder to another location.
-- Synchronize the now empty Dropbox folder with the iPad. Ensure that it is still empty after you're done, that nothing has re-appeared from the iPad. If anything has, either delete it or move it to a non-Dropbox location.
-- Re-install iOS Scrivener on the iPad. Configure synchronization, but there shouldn't be anything there.
--- One by one, move your projects back to Dropbox, Sync to the iPad -- which *will* take some time, since it's the whole project -- then make a minor change and sync again. The second sync should be very fast. If it isn't, you've identified which project is the source of the problem.

Katherine


Hi Katherine,

Thanks for your help trying to resolve this.

I've been following all your instructions. I've reinstalled Scrivener on my iPad and restarted with an empty Dropbox folder. I've also ensured that every single Project file I have is free of conflicted files, the trash is emptied, and any time it identified a problem with the Binder on opening, that Project file has now been rebuilt.

I've then started copying the files one-by-one, doing sync tests every so often (by changing one word and resyncing with Dropbox).

What happens is that the more Project files that get added to the DropBox folder, the longer "Downloading file list" is taking - this is AFTER the initial sync to download the files to the device. I can see that one progressing with x hundred files each time, which isn't a problem since you'd expect to wait for that to transfer.

It's subsequent syncs that are problematic. "Downloading file list" is slow even when the initial check has been performed and only one Project file (and I've been testing this on really small files) is identified as being problematic.

The time appears to be gradually increasing the more Project files I add back to the Dropbox folder. It hasn't suddenly leapt up from zero to 100 after the addition of only one file (so I have been unable to locate a single "problem" file). However, adding a large project file does seem to increase the time more when I try to sync the (tiny) test file.

* with 1-5 Project files in the Dropbox folder, syncing a single Project file takes about 5-10 seconds
* with ~25 Project files in the Dropbox folder, syncing a single Project file takes about 30 seconds
* with ~40 Project files in the Dropbox folder, syncing a single Project file takes about 55-60 seconds

As a final test, I've tried timing the update time with changing one file in a small Project (containing 1 document according to Statistics) and then changing one file in a large Project (containing 463 documents according to Statistics) and it took the same time - 60 seconds - to check and update them.

Can I suggest you try seeing what your sync time is like with maybe 20 Project files including some larger ones, and see if you notice a difference than with 5 Project files?

It really does seem that the presence of *other* Project files is affecting the checking and syncing of a *single* Project file, and this clearly wasn't the case previously (or if it was, it was very quick). As mentioned, I've been using this app since release, with dozens of Project files, and no problem. This is a new problem. It wasn't happening before.

I've also tried this over several different WiFi networks as well as wireless (my 4G connection is is actually much faster than my home landline broadband).

So, to sum up: the factor which appears to increase syncing time (in terms of "Downloading file list") is the number of Project files in the Dropbox folder, not the size of any individual Project file being synced.

I hope this helps pinpoint the issue.

User avatar
lunk
Posts: 2831
Joined: Wed Aug 21, 2013 4:24 pm
Platform: Mac + iOS
Location: Sweden 64° N

Sat Dec 16, 2017 11:32 am Post

If you have a lot of projects in the sync folder you fill up the memory in your iPad as well, which might also slow things down.

My solution is to have a Scrivener Projects folder in Dropbox, which has a subfolder named Active Projects which is the one I sync with the iPad. When I want to be able to work on a project on the iPad I use Finder on the Mac and simply move the project from the parent folder to the Active Projects subfolder. That way the iPad only has to sync 4-5 projects at the most. Hence, no slow down of sync times.
I am a user, writing non-fiction and science, using:
* Mac Scrivener 3 on a Macbook 12”, MacBook Pro 13”, and iMac 27”, all running the latest MacOS
* iOS Scrivener 1 on an iPhone 8, iPad Air 9.7”, and iPad Pro 12.9”, all running the latest iOS

is
istara
Posts: 65
Joined: Sat Nov 04, 2006 7:07 pm

Sat Dec 16, 2017 12:58 pm Post

lunk wrote:If you have a lot of projects in the sync folder you fill up the memory in your iPad as well, which might also slow things down.


I can't see how this could be the case, since it was fine before. The iPad I use is the 10.5" iPad Pro (not sure of RAM, but with 256 GB total memory, of which 184 GB is currently free) which I bought earlier this year.

I previously used Scrivener iOS on the iPad Air 2 - which I assume would have less RAM and has only 128 GB space, with 74 GB free)- and still never encountered this issue.

si
sidderke
Posts: 104
Joined: Thu Dec 03, 2015 11:34 pm
Platform: Mac

Sat Dec 16, 2017 3:36 pm Post

Isn't this related to this thread (where I just posted as well) :

viewtopic.php?f=53&t=49779&p=255162#p255162

?

It's a bit weird that a lot of people from support are saying "You have too much files in your Dropbox, your projects are too big", while this wasn't a problem before upgrading my projects to Scrivener 3.0, which I thought was gonna make most stuff quicker and more modern.
I have heard Scrivener 3.0 doubles the amount of index files, but it seems like there is more then double the amount of waiting. It's like it hits a bug or problem, once you have a certain amount of files, it goes *MUCH* slower instead of just double the amount of time as before.

User avatar
kewms
Posts: 4124
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sat Dec 16, 2017 5:22 pm Post

Scrivener 3.0 should lead to significant performance improvements relative to Scrivener 2 ON THE MAC.

iOS is another matter. Remember, Scrivener 3 is not running on the iOS device. There, the biggest contributor to sync time is the size and complexity of the project file. Yes, the Scrivener 3 project format is more complex and therefore more time consuming than the Scrivener 2 format, primarily because it is designed with the need to avoid and manage sync conflicts very firmly in mind.

It is not surprising that 40 projects with hundreds of files each take longer to sync than five projects with maybe a few dozen files.

I don't think anyone from L&L has said "your projects are too big," at least not that I've seen. It's a valid point, though. You are describing a "laptop replacement" use case, which isn't really what the original design of iOS Scrivener had in mind.

Katherine
Scrivener Support Team

si
sidderke
Posts: 104
Joined: Thu Dec 03, 2015 11:34 pm
Platform: Mac

Sat Dec 16, 2017 8:46 pm Post

Well, maybe I should be more nuanced: support told me trough email that it cannot be avoided if the number of files are big, like in the case of big projects or a lot of projects.
It's just weird that this wasn't a problem in Scrivener 2.8. I understand that if it has double the amount of files to sync, that it takes longer. I just don't understand why the downloading the file list doesn't take twice or even three times as long, if it's just that the number of indexing files doubled. It suddenly takes a lot a lot more time. I cannot measure it anymore, because I've updated all my projects, but it was seconds before, and it's about 5 minutes now. That's again, just something I don't understand, and worthy for development to check out. Did we hit a 'maximum' number of files suddenly that has Dropbox acting weird? I'm trying to understand the problem.

And I'm not really using iOS necessarily as a laptop replacement, but as a place to continue working on the projects, that are research-heavy. So for every small text that I change, I have to wait at least 5 minutes before it syncs. Syncing itself goes just as quick as before. It used to be: quick sync, just a couple of meta data things to change, because I would read research and change it's label. It all used to take seconds, and now takes minutes. If you have to do this for every device, it quickly adds up.

EDIT: if you look at the thread I linked to, user Mijalce is posting the same experience: In his case, his one project that wasn't too big, went from 12 seconds of syncing to 3.5 minutes.
So it's not just that the amount of files to be indexed doubled, resulting in double the amount of time to download the file list. It's like a 12x as long process (1200 procent increase) after updating to a format that should be more in line with the iOS format (because the iOS format was already ready for the 64bit codebase and 3.0 format).