Failed to load web archive error

User avatar
AmberV
Posts: 24184
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Ourense, Galiza
Contact:

Tue May 14, 2019 2:33 pm Post

Just to be clear, nothing is “lost”, and there is no need to recreate anything at all. There are ample methods for viewing your data if need be. What we are seeing here is merely a matter of the built-in viewer having been broken in an OS update. It requires a patch to make it functional again since Apple is never going to fix it; can’t say when that will be available yet, though.

  • On a one-by-one basis: hit ⌃⌘O (or click the little open-in-external-editor button in the footer bar). WebArchive opens in a window using Safari. Net result: much like using Quick Reference—or viewing any type of material you put into Scrivener that it itself cannot view in the main editor. It’s the same thing you’ll be doing for an Excel spreadsheet, or even a PDF or image if you wish to do more with it than just look at it.
  • Bulk use: select all needed files in the binder and press ⇧⌘E (or use File ▸ Export ▸ Files...). You will be asked to name a folder on the disk, into which the .webarchive files will be placed. These files can now be more easily managed with Finder, or with other utilities that support them.
  • [Destructive] Convert to industry standard text: select the .webarchive(s) in the binder and use the Documents ▸ Convert ▸ Web Page to Text menu command. As noted, this is a one-way process. If you wish to keep the originals, then duplicate a set of copies and run the command on the duplicates.

⠂─────── ⟢⟡⟣ ─────── ⠂

Going forward, and to digress a little, I’d give considerable thought to archiving material as text instead of in a proprietary single-operating system format like WebArchive. PDF is also a good alternative, especially if for some reason the presentation of the content is of importance—some sites even have a function for doing that right in the page (like Wikipedia). With Scrivener you can enable text archival in the Sharing: Import preference pane, with the Convert imported WebArchives and web pages to text setting.

At the very least, you will be saving yourself from actual loss and having to recreate everything, should Apple ever discontinue support in the next 40 years, or if for whatever reason you find yourself having to migrate to another operating system.

I’ve never used this format myself, the above being the strongest reasons for why. I also have concerns about how good it is for actual archival since it leaks to the Internet like a sieve when you view them. That also raises privacy concerns. If so much as having your Preview column open in Finder means tracking and advertising data is being sent from your computer to all of these sites, unimpeded by any of the browser extensions you may typically employ to protect yourself, that’s a problem. And since it is, in some regards, an “archive”, it is saved under whatever protections the law afforded you at that time, but not under whatever laws protect you now. A page from 2007 is not going to adhere to the GDPR. There are also security concerns when you have ancient site code being executed like that.

Sorry for the semi-off-topic rant, but in my book any time there is a glitch with WebArchive is a good time to point out its numerous flaws, as one has time to consider whether or not its a good format to continue using, when it stops working.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

User avatar
gavingough
Posts: 5
Joined: Thu Apr 18, 2019 8:13 pm
Platform: Mac
Contact:

Tue May 14, 2019 8:57 pm Post

Thanks for taking time to write about the flaws with the web archive.

A quick thought in response: would it not be better to remove that option from Scrivener if it's so troublesome? I've found that archiving web content with Evernote's web clipper and then adding links to the relevant Evernote notes or notebooks serves we well. Might that not be a more robust solution in the long term - and, given that the web archive isn't currently available in Scrivener, in the short term too?

User avatar
AmberV
Posts: 24184
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Ourense, Galiza
Contact:

Tue May 14, 2019 11:04 pm Post

I would not go so far as to say that on the whole it is troublesome, and many of the concerns I raised above are simply not that important to everyone. But they may be to some, and among those there may be those who didn’t realise Web Archive actually isn’t very secure at all and can even fail to render properly when fully disconnected from the ’net, which for them might defeat the purpose.

Situationally, long-term archival may not be something that matters to you—the convenience of having a quasi-offline article for a project that isn’t going to be opened ever again after you publish in three months is probably greater than any librarian level memorial to the resource. Likewise there may be those that really don’t care if Google is constantly crooning “I’ll be watching you…” into their ear as they go through the day. Some people even use Android phones for that matter. :lol:

It’s less our place to police what people can do, is what I’m getting at. I’m more a fan of leaving the options open, because we cannot anticipate what everyone wants or needs of a tool, than to say that because I personally would avoid this tool, everyone else should too. For myself, I prefer long-term “memorial” archives—so I use text. Civilisation will have to collapse before I lose access to that data.

My point above is better stated as such: it is moments like this when some single-point tech fails that we should question whether or not that’s the best tool for something that needs be reliable. It’s a response to those who find themselves in a snarl with no backup plan (though as I point out, I think they have better options than they think they do). And to be clear, sometimes it still is the best answer, even with a single point failure. In this case, even though the viewer is offline, the data is still accessible, as demonstrated by the continued functioning of the conversion to text feature.

On the matter of Evernote, I don’t use it, so I’m not entirely sure what all the web clipper is or does, but it sounds like it is doing something similar to Web Archive? I don’t so much have a natural distrust of this general concept. There are formats and tools that actually do full downloads where you will not be sending out pings to advertisers and no part of the page will be missing that you might have depended upon. These formats tend not to be as “convenient” however—they are usually thought of more as site clones, where you end up with whole directories of files on your drive—basically a copy of every part of the website necessary to display the page, from the scripts to the fonts to the stylesheets to the content. Something like that would not represent as well, or as simply, in the Binder I suspect.

Regarding Evernote pages in .webarchives though—not sure if that is what you’re asking about, but I’d try turning off your WiFi antenna, or unplugging the ethernet, and seeing if you can access the notebook you saved into the binder. It’s been several years since I’ve seen anyone mention doing that, but I recall back then the content itself wasn’t archived, only the container around it. You had to be online to view the content—which meant it wasn’t really an archive in the snapshot sense of the term. If the notebook was deleted or mangled online, you’d lose the data in Scrivener too. So if it is your intention for this thing to be a secure backup, I’d make sure it actually is.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

User avatar
gavingough
Posts: 5
Joined: Thu Apr 18, 2019 8:13 pm
Platform: Mac
Contact:

Tue May 14, 2019 11:42 pm Post

Thank you for taking the time and trouble to share your thoughts in more detail. Somewhat uncharacteristically, I am in full agreement.

In a former and less glamorous incarnation as a Systems Analyst, concerns about data security used to keep me awake at night. Or, at least, they might have done if I'd taken the position seriously. But that's another story.

I frequently work without a wifi signal or ethernet connection so need access to locally stored research. Evernote's web clipper extracts text from web pages in various formats. There's a full page option, similar to the Web Archive (but more robust), a simplified, streamlined option for storing text without formatting, a bookmark option and a screenshot option. I take screenshots or extract the juicy text.

I shuffle these into 'notebooks', tag them with meaningful tags and the content is stored offline. For me, it's proved to be a robust way to store and organise my research. I can append comments, add highlights, merge notes and can refer to my research even when civilisation collapses, providing I can find my spectacles.

I tend to have Scrivener and Evernote open together and that's proved to be an effective way to work.

Disclaimer: I'm not on commission, don't work for Evernote and my mother isn't Evernote's CEO. It is one of those increasingly rare applications that, like Scrivener, does what it's supposed to do, doesn't demand too much but gets out of the way and allows me to work without fuss and, to the best of my knowledge, without telling Google that I've recently purchased a pair of handsome brogues and a jaunty panama hat.

Although now I've included that information in this post, I suppose the secret is out and I'll see advertisements for gent's outfitters at every turn.There really is no escape!

Pe
Peyton Stafford
Posts: 3
Joined: Sat Apr 13, 2019 3:56 pm
Platform: Mac

Fri May 24, 2019 3:35 pm Post

I found a workaround for this error. After you save the web page in Scrivener, go to the Documents menu > Convert > Web Page to Text. Scrivener will give you a document that may be badly formatted, but at least it's something to make notes from, and it moves the original URL to the Document Bookmarks.

go
goshng
Posts: 3
Joined: Sun Feb 14, 2016 11:36 pm
Platform: Mac

Mon May 27, 2019 2:13 am Post

I would use the feature of text document conversion when importing a website because Apple would never fix the issue. Then, I wonder if ^+Command+O shortcut can be used to open the original URL for a text document converted from Web Archive. Thanks.

User avatar
AmberV
Posts: 24184
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Ourense, Galiza
Contact:

Mon May 27, 2019 11:53 am Post

You can click the link in the footer bar to load the present-tense page in a browser. The “Open in External Editor” shortcut does just that: it opens the archived file (typically also into a browser). So which to use depends on whether you want to see the page as it is now, or as it was when you archived it.

@Peyton: regarding the work-around, I posted this one and a number of others here, a few posts back.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

li
lindywarrell
Posts: 68
Joined: Fri Nov 27, 2009 9:50 am
Location: Australia

Tue May 28, 2019 12:38 am Post

I'm so glad I finally found this second 'thread' in the issue of lost web files. Who knew? Nobody replied to me when I last posted question about it so I found this reference to a second thread (this one) quite by accident. Anyway, it serves my purposes for now.

AmberV wrote:Just to be clear, nothing is “lost”, and there is no need to recreate anything at all. There are ample methods for viewing your data if need be. What we are seeing here is merely a matter of the built-in viewer having been broken in an OS update. It requires a patch to make it functional again since Apple is never going to fix it; can’t say when that will be available yet, though.

  • On a one-by-one basis: hit ⌃⌘O (or click the little open-in-external-editor button in the footer bar). WebArchive opens in a window using Safari. Net result: much like using Quick Reference—or viewing any type of material you put into Scrivener that it itself cannot view in the main editor. It’s the same thing you’ll be doing for an Excel spreadsheet, or even a PDF or image if you wish to do more with it than just look at it.
  • Bulk use: select all needed files in the binder and press ⇧⌘E (or use File ▸ Export ▸ Files...). You will be asked to name a folder on the disk, into which the .webarchive files will be placed. These files can now be more easily managed with Finder, or with other utilities that support them.
  • [Destructive] Convert to industry standard text: select the .webarchive(s) in the binder and use the Documents ▸ Convert ▸ Web Page to Text menu command. As noted, this is a one-way process. If you wish to keep the originals, then duplicate a set of copies and run the command on the duplicates.

⠂─────── ⟢⟡⟣ ─────── ⠂

Going forward, and to digress a little, I’d give considerable thought to archiving material as text instead of in a proprietary single-operating system format like WebArchive. PDF is also a good alternative, especially if for some reason the presentation of the content is of importance—some sites even have a function for doing that right in the page (like Wikipedia). With Scrivener you can enable text archival in the Sharing: Import preference pane, with the Convert imported WebArchives and web pages to text setting.

At the very least, you will be saving yourself from actual loss and having to recreate everything, should Apple ever discontinue support in the next 40 years, or if for whatever reason you find yourself having to migrate to another operating system.

I’ve never used this format myself, the above being the strongest reasons for why. I also have concerns about how good it is for actual archival since it leaks to the Internet like a sieve when you view them. That also raises privacy concerns. If so much as having your Preview column open in Finder means tracking and advertising data is being sent from your computer to all of these sites, unimpeded by any of the browser extensions you may typically employ to protect yourself, that’s a problem. And since it is, in some regards, an “archive”, it is saved under whatever protections the law afforded you at that time, but not under whatever laws protect you now. A page from 2007 is not going to adhere to the GDPR. There are also security concerns when you have ancient site code being executed like that.

Sorry for the semi-off-topic rant, but in my book any time there is a glitch with WebArchive is a good time to point out its numerous flaws, as one has time to consider whether or not its a good format to continue using, when it stops working.

dr
drmajorbob
Posts: 318
Joined: Sun Jul 01, 2012 1:38 am
Platform: Mac

Sun Jun 09, 2019 6:50 pm Post

I heard rolling back to a previous Scrivener version could fix it. Is that possible?

User avatar
AmberV
Posts: 24184
Joined: Sun Jun 18, 2006 4:30 am
Platform: Mac + Linux
Location: Ourense, Galiza
Contact:

Mon Jun 10, 2019 12:22 pm Post

That wouldn’t work because the problem is not in Scrivener. You would have to roll back to macOS 10.14.3, and then everything would work as expected again.
.:.
Ioa Petra'ka
“Whole sight, or all the rest is desolation.” —John Fowles

ar
arcy
Posts: 2
Joined: Sun Jun 16, 2019 1:16 pm
Platform: Mac

Sun Jun 16, 2019 1:27 pm Post

This situation, and the company's response, are both highly disappointing and exasperating.

If I had an iOS app that integrated with, say, the Facebook API, and then Facebook changed their API in a way that broke my app, would I say my broken app is Facebook's fault? NO WAY. If I make an app that integrates with the Facebook API, it is my job to update my app in line with the changes of the Facebook API. Since I chose to have my app depend on that system, that is part of my responsibility.

However, here, L&L is passing the buck to Apple.

If you make Mac software, it is your responsibility to your paying customers to keep updating the code so that the functionality persists.

Something IS lost here. The "Research" functions of Scrivener are not bells & whistles; the ability to add to and browse through research culled from the Internet in a single UI, ideally whether online or offline, is the main reason many of us keep using Scrivener. There are ample replacements out there for distraction-free writing and project organization these days.

If the tune doesn't change here, I guess I'll switch to one of Scrivener's many competitors for distraction-free writing and project organization, and try building a modern cross-platform open-source research-gathering tool myself. Any other coders want to join me?

User avatar
gavingough
Posts: 5
Joined: Thu Apr 18, 2019 8:13 pm
Platform: Mac
Contact:

Sun Jun 16, 2019 2:26 pm Post

Back in March, when the issue first became clear, I think it was suggested that users try DevonThink, KeepIt and EagleFiler to see if their webarchive functionality was working.

I've followed the suggestion and tried all three and can save and access web archives. As far as I can see, only Scrivener is unable to display web archives as expected.

Given the caveats already discussed and considering the length of time that this has been an issue, it doesn't seem unreasonable to expect the issue to be resolved now. Or, if that's not going to be possible, for the web archive option to be removed from Scrivener.

Online
User avatar
kewms
Posts: 6049
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sun Jun 16, 2019 4:10 pm Post

I believe part of the issue has been the sudden and unexpected need to implement new licensing functionality, as discussed here:
https://www.literatureandlatte.com/blog ... ith-paddle

For obvious reasons, that has jumped ahead of all other Mac development on the priority list.

Katherine
Scrivener Support Team

User avatar
lunk
Posts: 3970
Joined: Wed Aug 21, 2013 4:24 pm
Platform: Mac + iOS
Location: Sweden 64° N

Sun Jun 16, 2019 5:16 pm Post

arcy wrote:The "Research" functions of Scrivener are not bells & whistles; the ability to add to and browse through research culled from the Internet in a single UI, ideally whether online or offline, is the main reason many of us keep using Scrivener.
- - -
If the tune doesn't change here, I guess I'll switch to one of Scrivener's many competitors for distraction-free writing and project organization, and try building a modern cross-platform open-source research-gathering tool myself.

How many? One per cent or ten?

I am looking forward to seeing the Arcy app. :D
I am a user, writing non-fiction and science, using:
* Mac Scrivener 3 on a Macbook 12”, MacBook Pro 13”, and iMac 27”, all running Mojave.
* iOS Scrivener 1 on an iPhone 11 Pro, iPad Air 9.7”, and iPad Pro 12.9”, all running the latest iOS

Online
User avatar
kewms
Posts: 6049
Joined: Fri Feb 02, 2007 5:22 pm
Platform: Mac

Sun Jun 16, 2019 5:29 pm Post

arcy wrote:Something IS lost here. The "Research" functions of Scrivener are not bells & whistles; the ability to add to and browse through research culled from the Internet in a single UI, ideally whether online or offline, is the main reason many of us keep using Scrivener. There are ample replacements out there for distraction-free writing and project organization these days.


FWIW, there are many different ways to capture research from the Internet, and the web archive format has always been one of the least robust. It's certainly not accurate to say that problems with that format alone mean that the Research functions of Scrivener have been "lost."

Katherine
Scrivener Support Team