Suggested spellings no longer function under Mojave

wa
warrenao
Posts: 37
Joined: Sat Feb 09, 2013 8:52 pm
Platform: Mac

Fri Sep 28, 2018 8:00 pm Post

This is happening to me, at any rate (Scriv 3.0.3). Used to be I could right-click a misspelled word and select the suggested correction from a popup. This was really useful on words I wasn't sure of, such as "triskaidekaphobia."

Now, however, since upgrading to Mojave, I never get spelling suggestions, even on really easy words. Here, I've deliberately mistyped "therapist". It's been correctly flagged as misspelled, but there are no suggestions to correct it.

Screen Shot 2018-09-28 at 12.57.15.png
Screen Shot 2018-09-28 at 12.57.15.png (36.44 KiB) Viewed 6593 times

User avatar
kewms
Posts: 6447
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Fri Sep 28, 2018 11:42 pm Post

Please test this in TextEdit. It is probably an issue with the Mac OS spellcheck system, rather than Scrivener itself.

Katherine
Scrivener Support Team

wa
warrenao
Posts: 37
Joined: Sat Feb 09, 2013 8:52 pm
Platform: Mac

Sat Sep 29, 2018 12:55 am Post

kewms wrote:Please test this in TextEdit. It is probably an issue with the Mac OS spellcheck system, rather than Scrivener itself.

Katherine


It would seem not, alas:

Screen Shot 2018-09-28 at 17.53.56.png
Screen Shot 2018-09-28 at 17.53.56.png (46.88 KiB) Viewed 6577 times

wa
warrenao
Posts: 37
Joined: Sat Feb 09, 2013 8:52 pm
Platform: Mac

Sat Sep 29, 2018 1:12 am Post

Also, FWIW, this is happening on both of my Macs, a Mini 2012 and an Air 2014. Both are running Mojave and Scriv 3.0.3, and neither showed this behavior prior to the OS update.

wa
warrenao
Posts: 37
Joined: Sat Feb 09, 2013 8:52 pm
Platform: Mac

Sat Sep 29, 2018 2:22 am Post

This is interesting. Following the suggestion posted here https://www.macobserver.com/tmo/article/os-x-fixing-autocorrect-issues, (a MacObserver article on customizing the user spelling dictionary), I tracked my own dictionary down.

Its contents are indeed a list of my custom entries, some of them years old — but as you can see from the file create/modify date, it's fresh, dating back to the date and time I installed Mojave, as it happens.

Since then, it apparently hasn't been touched by the filesystem, and I know I've added custom words since then. Where they've ended up instead is the file just above it, the one named "en". Presumably as in "english". It has a creation/mod date of just a few minutes ago.

I'm not sure this is related, but it's an interesting situation, and it does have to do with spelling. Has Apple changed how spelling dictionary customizations work on the back end?

Screen Shot 2018-09-28 at 19.10.03.png
Screen Shot 2018-09-28 at 19.10.03.png (256.97 KiB) Viewed 6568 times

User avatar
KB
Site Admin
Posts: 20775
Joined: Tue Jun 13, 2006 11:23 pm
Platform: Mac
Location: Truro, Cornwall
Contact:

Sat Sep 29, 2018 10:04 am Post

Curious! I've checked the code, and it's all good (and quite simple): it first asks Apple's spell checker (NSSpellChecker) to see if there is a misspelled word at the selected range and then calls the spell checker's -guessesForWordRange:... method, which returns a list of suggestions. Scrivener then lists those or shows "No guesses found" if that returns nothing. So for you to be seeing no guesses, Apple's own spell checker has to be returning on guesses for the word.

I did some testing myself and found odd results. At first, everything worked fine. Then I used "Learn '[misspelled word]'" on a word, and after that I received no suggestions for "therpist". When I quit and restarted Scrivener, the suggestions reappeared. I tried this several times and both "Learn" and "Unlearn" seemed to break things. However, then things started working again, randomly - even using learn and unlearn didn't break things and my suggestions worked fine every time. So I booted into a clean account, and this time everything worked without any problems.

So right now, even though I've seen it happen, I'm afraid I cannot break it consistently, which means that I can't report it to Apple yet as I have no perfect reproduction case (and this is almost certainly an Apple bug since it's their spell-checker). The only thing I can think of is that maybe the text view is reporting a wrong selection range so that the wrong text is getting sent to the spell-checker. Hmm.
"You can't waltz in here, use my toaster, and start spouting universal truths without qualification."

wa
warrenao
Posts: 37
Joined: Sat Feb 09, 2013 8:52 pm
Platform: Mac

Sat Sep 29, 2018 3:33 pm Post

KB wrote:I did some testing myself and found odd results. At first, everything worked fine. Then I used "Learn '[misspelled word]'" on a word, and after that I received no suggestions for "therpist". When I quit and restarted Scrivener, the suggestions reappeared. I tried this several times and both "Learn" and "Unlearn" seemed to break things. However, then things started working again, randomly - even using learn and unlearn didn't break things and my suggestions worked fine every time. So I booted into a clean account, and this time everything worked without any problems.


Curious is right. I've done some poking too, since reading your reply. The file here called dynamic-counts.dat got updated when I exited Scriv. TextEdit apparently can open it, but only shows what looks like a hash. However, when I open in in TextWrangler, I see a properly-formatted dict plist, with key and integer pairs, all of which appear to refer to nothing but different language dictionaries. (de, en, en_US, etc).

The write date/time on the file called "en" hasn't changed, nor has "LocalDictionary".

I created my own test account, and when I went to its ~/Library/Spelling folder, found no documents in it at all. Not even a LocalDictionary, en, or dynamic-counts document. So going back to my own library/spelling folder, I began poking around more.

The file called dynamic-text.dat shows a mod date of 8 July, which is some old-ish stuff. Inside it, I found a collection of misspelled words, words which I've mistyped myself (typically transposed keystrokes), and it's clearly based on words I've written. There are, for instance, mistyped character names from stories in there.

dynamic-text-tmp.dat is like that too, but a much shorter collection. Same creation and modification date and time as dynamic-text.dat.

LocalDictionary is showing no updates since three days ago, even after I've added new words to the spelling dictionary.

en is showing a mod date of yesterday, which is consistent with when I added new words. But "en" wouldn't be my spelling pref; it should be "en_US", I'd think. Maybe not.

So I figured why not, duplicated the contents of my user Spelling folder to my desktop, and nuked the documents in ~/Library/Spelling. I reloaded Scriv, told it to learn the word "therpist" (there were no suggestions in the popup), and a new "en" document was created, with that one word in it. When I told Scriv to unlearn the word, "therpist" got entered a second time in "en".

When I exited Scriv, there were no other new documents in ~/Library/Spelling, and "en" still contained both entries of "therpist".

There is no new "Spelling" folder being kept on the system level (in the top-level /Library folder, for instance). It's still only local user storage.

I wondered if copy/pasting the contents of LocalDictionary to en would preserve all the words I told this machine to learn, down through the years. (There are more than 2500 entries.) So I did that. I re-tried the "therpist" gambit, and when I added it to the dictionary, it appeared in the "en" list, along with the other 2500+ entries. This time when I told Scriv to forget it (there were still no suggestions in the popup, though they're appearing elsewhere), the misspelled word was deleted from the "en" dictionary file.

So … has Apple changed the nomenclature for user spelling dictionaries from LocalDictionary to [language]? And, if so, is Scriv's code looking for the correct file now? Or is it still looking for the old "LocalDictionary" file? Might that be doing something to prevent suggestions from appearing in the popup?

Screen Shot 2018-09-29 at 07.28.51.png
Screen Shot 2018-09-29 at 07.28.51.png (135.94 KiB) Viewed 6531 times

wa
warrenao
Posts: 37
Joined: Sat Feb 09, 2013 8:52 pm
Platform: Mac

Sat Sep 29, 2018 4:09 pm Post

To clarify: "LocalDictionary" stopped being modified on the day I installed Mojave. Since then all I've seen are entries to "en".

Character names (e.g., "Lithros") entered in my original LocalDictionary are still recognized as valid after copy/paste into "en". LocalDictionary is no longer resident in my ~/Library/Spelling folder.

Best surmise is "LocalDictionary" has been obsoleted in favor of [language] for the user spell dictionary.

Meanwhile, spelling suggestion seems to be dependent on the length of the document. In a story file with 11K+ words, suggestions don't appear in the popup. In a file with only a few dozen words, I get suggestions in the popup.

User avatar
KB
Site Admin
Posts: 20775
Joined: Tue Jun 13, 2006 11:23 pm
Platform: Mac
Location: Truro, Cornwall
Contact:

Sat Sep 29, 2018 4:38 pm Post

To be clear, Scrivener isn't looking for any files at all, let alone obsolete ones. Scrivener does nothing but call Apple methods. Here is the entirety of the code that learns a word:

[[NSSpellChecker sharedSpellChecker] learnWord:word];

And here is the entirety of the code that looks up the suggestions:

NSArray *guesses = [[NSSpellChecker sharedSpellChecker] guessesForWordRange:selectedRange inString:string language:nil inSpellDocumentWithTag:[self spellCheckerDocumentTag]];

NSSpellChecker is Apple's object that handles spell-checking. So Scrivener is doing nothing wrong at all here - anything going wrong is going wrong in Apple's NSSpellChecker code - and that's a black box, unfortunately, so there's no way for me to modify it. The best I can do is try to find a reproduction case - a case where the guesses *always* stop working - and report it to Apple so that they can fix it. But unfortunately at the moment I cannot find a case that always breaks things. Can you find a method in a clean account of getting it to break every time?
"You can't waltz in here, use my toaster, and start spouting universal truths without qualification."

wa
warrenao
Posts: 37
Joined: Sat Feb 09, 2013 8:52 pm
Platform: Mac

Sat Sep 29, 2018 5:43 pm Post

KB wrote:Can you find a method in a clean account of getting it to break every time?


Thanks for the look under the hood.

Alas, I don't seem to be able to make it break at all, in a clean account. I generated 2500 words of Lorem Ipsum (http://www.blindtextgenerator.com/lorem-ipsum) and pasted that, multiple times, into a document until it was at 15K words. I still got suggestions, even after adding some bogus words to the user dictionary. Even after exiting and re-loading Scrivener, I still got suggestions.

I also duplicated that document multiple times in Drafts, and still got suggestions. Even after I changed the typeface and line spacing, I was getting suggestions properly.

However, I have a bit more to report in my old moved-in standard user account.

In the document I first noticed the troubles with, it's only the main editor window where suggestions fail. In the Notes pane, misspelled words are highlighted, and the popup works with suggestions there. In the main editor window, the exact same word will highlight as misspelled, but I get no suggestions. There are only 150 or so words in the Notes pane. There are 11K+ in the main editor. That was why I was thinking it had to do with word count — particularly as a new document in the same project does not show this behavior. The main document window will provide suggestions in the popup, when there are a few dozen words in it.

When I pasted LI text into a new blank file in my current pre-existing project, the suggestions worked at 5000 words, but failed to appear at 7500. I honed the copy/paste until I found the cutoff; at 6900 words, I got suggestions in the popup. At 7000, the suggestions were no longer present. But that was not in a new, fresh user account, or in a new, blank project.

The other thing I've noticed is there is a distinctive lag between my right-clicking a word, and the popup appearing, when the auto-suggestion fails.

Total word count in the existing project is… well, here:

Screen Shot 2018-09-29 at 10.42.28.png
Screen Shot 2018-09-29 at 10.42.28.png (52.28 KiB) Viewed 6520 times

wa
warrenao
Posts: 37
Joined: Sat Feb 09, 2013 8:52 pm
Platform: Mac

Sat Sep 29, 2018 5:54 pm Post

One more thing. When I opened the "Text" tab under the "Keyboard" system prefs, I found I'd set my spelling preferences to "US English" at some point in the past. Changing it back to "Automatic by Language" resulted in the creation of a new empty "LocalDictionary" file in ~/Library/Spelling, alongside the "en" file.

So that's clearly unrelated to Scrivener, and I apologize for bringing it up. I think the contents of the ~/Library/Spelling folder can be ignored here.

Fl
Fletcher DeLancey
Posts: 14
Joined: Sun Dec 18, 2011 6:27 pm
Platform: Mac

Sat Oct 20, 2018 11:17 pm Post

I'll jump in on this with a related issue: my custom spellings (of which there are many, as I have quite a few invented words and names in my fiction projects) were no longer being recognized. Following the discussion above, I checked out my Spelling folder and found the two indicated files, LocalDictionary and en. LocalDictionary was empty; en contained my list of custom words and names. But they were not being referenced. When I opened an existing project to edit it, I saw a sea of red underlining.

As a test, I used the Learn command for a handful of these already-learned words, then rechecked the Spelling folder. All of those newly learned words were now in the previously empty LocalDictionary file. This seems to be precisely the opposite of what happened to Warrenao.

At any rate, copying the contents of en into LocalDictionary and restarting the computer seems to have fixed the issue.

That said, there are still a few red-underlined words in my text that I'm certain were in my custom dictionary earlier. Apparently they fell through the cracks during the transition from High Sierra to Mojave.

wa
warrenao
Posts: 37
Joined: Sat Feb 09, 2013 8:52 pm
Platform: Mac

Sat Oct 20, 2018 11:36 pm Post

Sounds well familiar, Fletcher. I'd specified English as my dictionary under Keyboard Preferences (Apple system prefs pane), and that was what created the "en" document in my ~/Library/Spelling folder. When I switched those prefs to default, LocalDictionary came into existence. I've left it at the default since then. Copy/paste of the contents of "en" populated the dictionary for me, which was convenient.

The red underlines appear to be a side-effect of Mojave. Its grammar checker keeps intruding into places it has not been invited to. I believe L&L is working on a fix for that now, creating a workaround to do what Mojave should be smart enough to do on its own.

I've found with high consistency that any document containing 7.5 - 8K words or more no longer gives spelling suggestions in the popup. But if it's just a few hundred words, or even, say, 6000, suggestions work as expected.

Also, FYI, you don't need to reboot after manually changing the contents of your spelling document on the HD. You can open Terminal and enter these commands:

Code: Select all

launchctl stop com.apple.applespell [enter]
launchctl start com.apple.applespell [enter]


That painlessly unloads, then reloads, Apple's spell service. (The entire reason I knew that is because I didn't want to have to go through a 2-minute reboot once, so spent 15 minutes googling for workarounds, cause that's how I roll. :wink: )

Fl
Fletcher DeLancey
Posts: 14
Joined: Sun Dec 18, 2011 6:27 pm
Platform: Mac

Sun Oct 21, 2018 12:31 pm Post

That painlessly unloads, then reloads, Apple's spell service. (The entire reason I knew that is because I didn't want to have to go through a 2-minute reboot once, so spent 15 minutes googling for workarounds, cause that's how I roll. )


Ha! I get that -- I recently had the bright idea of using Typinator + Regex to automate something I do with great frequency (replace one carriage return with two for clipboard text), so I first spent two hours on Regex tutorials, then another half hour with the Typinator help documentation, then finally gave up and emailed the developer. He emailed back with a ridiculously simple bit of code that I'd never have figured out on my own (without taking a class in Regex). I figure I'll need to use that Typinator shortcut a few thousand times before the time savings makes up for all that research.

On the other hand, it's never a waste of time to use one's brain. Plasticity for the win!

Anyway, thank you for the excellent Terminal tip.

wa
warrenao
Posts: 37
Joined: Sat Feb 09, 2013 8:52 pm
Platform: Mac

Sun Oct 21, 2018 2:43 pm Post

Rather a bit on the same page, Fletcher, in analogous ways. It's astonishing sometimes how inefficient efficiency can be.

But, really, yes, it does boost neuroplasticity. It would have to. Digging out obscure OS secrets, RegEx calls, or crafty bits of code is another form of playing games, particularly logic puzzles.