Scrivener 3 didn't import my bookmarks from Scrivener 1

[simeva]

Just seen this post from Amber while composing this reply to Clive, to whom - thanks again for your help.

I thought I should understand more about the problem before tackling it. First I used Notepad ++ to establish the scale of the loss: 2770 bookmarks missing from 1344 files (“containers”) across 4 projects.

I then downloaded some XML editors to examine the .scrivx files, and realised I could adapt your suggestion of copying and pasting from within Scrivener to Microsoft XML Notepad. It is a simple matter to display the two .scrivx files next to each other as trees, search for each instance of the element in the old file, and then drag and drop each one across to the right place in the new file. This transfers all the bookmarks in that container in a single action. The imported elements then need to be renamed to conform to the new method described by AmberV above, and any duplicates merged.

It took me half a day to work out how XML works and set this up, and another half a day to fix one of my smallest projects. Once I was up and running, I repaired about 90 containers in an hour, by which time my eyes were so glazed over I had to go for a walk. I estimate that to fix all my projects it would take about 30 hours of continuous work, spread over at least 6 days. So: this is a solution, but still a painful one.

The good news is that the work process was simple and iterative, and should be reproduceable in software. Here is a sketch of a spec:

• The BAK file is my older Scrivener 1.9 XML back-up with the bookmarks in it.
• The LIVE file is my newer Scrivener 3 XML file.

1. Create a list of the missing bookmarks: BookMarkList.
   From the start of the BAK file,
   Find the next element (note the plural: References).
   Export the whole element (with its children), together with the UUID attribute of its parent element.
   Add it to the BookMarkList.
   Repeat until the end of the BAK file.

2. Import the missing bookmarks into the LIVE file.
   For each item on the BookMarkList,
   Find the element in the LIVE file with the matching UUID attribute.
   Import the whole element into that . (Does the exact ordering of the elements matter? I don’t know - I reproduced the order as it was in the BAK file.)

3. Update the element names.
   In the LIVE file,
   Replace ‘Reference’ with ‘Bookmark’ (not whole words only, so that the plural ‘References’ is replaced with ‘Bookmarks’)in the element names only (in case the word ‘refereence’ appears in a url or title).

4. Merge any duplicate elements.
   From the start of the LIVE file,
   Find the next that contain two elements (note plural: Bookmarks).
   Merge the two elements.
   Repeat until the end of the LIVE file.
   This step corrects any containers where the Scrivener update successfully transfered old internal references, and also where new references have been added since the update.

Isn’t that it? Would it really take several programmers more than a week to write that? I’d certainly be happy to help test it. It may be even simpler, if the import/update/merge operations were done on the fly, without creating an intermediate BookMarkList. But I find it clearer to explain it that way.

You mentioned that the nested structure of the XML file complicates this, but does it really, if approached in this way? And I can’t think of any scenarios where post-update edits to the LIVE file such as moving containers, adding text or metadata would trip this up. Maybe I’ve missed something.

I had a look at the scripting tools you mentioned to see if I might have a go at this myself, but sadly that is beyond me. I know a bit about logic, but not much about programming.

What do you think?

[simeva]

Thanks for your reassurance that you will look into this, Amber. It would have saved me a lot of worry if you had said so nearly four weeks ago when I first raised the issue.

[narrsd]

Simeva,

You are clearly a very capable person, at what one say is a professional level, and no matter that dealing with this is a bit outside your normal scope.

And yes, you are ‘pretty good’ with logic…and I imagine other things. I’m moved to include something I’ll put at the end, in case you may like it…

Now, briefly on your point about programs written. You’ve figured out for yourself the layout here, and that’s great. The thing about computers is that they are so very literal – other words have been used.

So everything you’ve mentioned, but much more besides that you’ve ‘just tone’ to recognize and adjust for anomalies has to be spelled out in a program. And recognized as a possibility before it can be so expressed. And tested against complex enough situations to have some sense that reality has been well enough expressed. And provided with all the niceties persons need and expect, when the coding reaches its limits, or otherwise is unable to fully perform.

I intentionally just write that as a mass, because that’s the way it is, the hidden aspect of the iceberg. To get the oceanic diving to occur where the full mass is found, you have to gain a person in the frame of mind to do it, in the studied intention to do it. This is a problem in working groups, and it’s a particular problem for the individual and person who actually has to engage in the work.

It’s unusual that they would have a sense of motivation as you’ve had, or the evident practice in carrying something out – and if they are able to, there’s a limit to how many problems they can fit in a day, a week, a year along with all the other rich aspects of life, so we are treading on grounds up towards novels here, aren’t we…

I’m going to leave this, then, and just say I have probably the perspective to know such problems, out of a not so short and mostly anthropologically composed life. Anything technical has just been a tool, if we inevitabbly have to keep picking up the sharp ones, as in your wit of approaching the situation here.

Now, I promised, so here is something, given it will open for you – if not, PM me, and I’ll send you a PDF I took while it was open.: Kazuo Ishiguro Interview

This is a remarkably open discussion with an author who has appeared to be, and in the sense of his writings, truly as hermetic aa can be.

Yet particularly if you may know well enough the British so that you recognise who’s talking as he speaks, or indeed something of Japanese persons, his constant stream of intentional disclosure for what he’s actually been and done, and as purposefully spoken in the very open , slang of times language which entirely suits it, I consider again this morning offer quite a set of doorways into what fiction writing can actually be.

In there too, some unadorned, succinct generosities, in how such things might be put together, with the hands…

a smile accompanies, as often
Clive

[narrsd]

Simeva (@simeva ), I’d been having a pretty philosophical day yesterday, but something in what you’d said stuck enough to make me go back and look again, for what I might have skipped over.

What I’d missed primarily is the amount of effort doing the initial run on your smaller project cost you.

Going over your procedure, I found myself today just nodding – the way you’ve arranged approach and details, you’ll have avoided any number of errors persons might generally make; it’s very good. So I don’t think I can argue you that XML is very critical to the last punctuation point, though you could have the entire file and thus your project fail by a slip, and that this could be pretty hard to find…

This is just one of the reasons, though, why I came to recommending the simplistic-seeming method of using two open Scrivener windows and a web browser visible at its address line to directly transfer bookmarks You’d just click one in the old project to open it in the browser, then drag-drop the address line into the new project on the same Scrivener page’s Inspector field.

I suspect that this has marked advantages.

• First, to address discomfort, this has to be a much less demanding and much more rapid set of moves than editing lines in and out of XML files. You will not have to work as hard!

• then, it has the less visible advantage that you can’t make mistakes. You’re not getting involved with the XML lines directly, but letting Scrivener handle all that…

• …which has another hidden advantage. XML is not only precise in its requirements for tiny textual elements, but also in the structure of the files.

  They have to be exact in many ways, to do with orderings, tags, and some of the extra attributes I noticed changed between the Scrivener versions, if those are used.

  So again, it’s very healthy indeed to let Scrivener do all the work properly for itself, working entirely from the link you dragged in.

• And what about the possible issues where you’ve worked ahead on the new project for a while? Where you’ve added bookmarks to it, you’d need to merge your old ones in, only the lines themselves, not duplicate the enclosing label lines

• Lastly, in this method you’ll also easily pick up any bookmarks which are not web links, and similarly can be sure to have their references come out correctly, by recreating them manually, worth the little effort not to assume old coded connections would be right.

Ok, and just one more point worth a bullet:

• I mentioned an even more relaxed and effective way to bring the bookmarks over: to not pre-work everything, but to just move in a section’s old bookmarks when you are actually working on it, and thus would have use for them.

  This might or not suit your mood about such things, to not be complete up front, I do appreciate, and that you’d work that out yourself

---

(this rest I just include for completeness, since I think you like to know the whole of a thing)

That does it for recommendations, but you might like to know I took one more pass over any sensible way to automate this, assuming the kinds of necessary details mentioned.

• That you mentioned XML editors sent me to look at any automation/scripting abilities these might have. Very primitive, is the answer, if useful for the sorts of awful tasks people who maintain big computer installations are sent to do, after this XML, which is designed only for machines, has been required by their managers, and inevitably needs adjustment.

  There are a lot of very confused questions out there on doing such things, and some very profiteering companies.

• one thing that did turn up, though, was a sensible, experienced person asking why not to use real programming to do the job, as it has the abilities, also showing a language and an effective helper library for it.

  Here we’re back to the answer you asked for, that Scrivener Windows get this corrective upgrade, especially as all the details and tools are known there, and of course you are correct to ask for it.

Thus we’re equally back to hoping that @AmberV and any others reading here will be able to have good influences.

Meanwhile, we do the practical things, don’t we, and I’m still well appreciative that you are, while hoping suggestions will make it easier.

Best, Simeva,
Clive

[simeva]

Thanks @narrsd for the link to the Ishiguro interview, which I do like - it’s candid, unpretentious, direct. It’s inspired me to start When We Were Orphans, which I have not read.

I prefer using the XML editors to move these bookmarks because:

• It’s less work: I only have to make one move for each Scrivener container, rather than one move for each bookmark. Where there are ten bookmarks on a page, this is a real help.
• Though there may be a danger of messing up the XML file, I do not have to write or copy text using the keyboard. It’s all drag and drop operations, and then the editor’s Replace function.
• The merging of new bookmarks is a bit fiddly, but again it’s only a drag and drop operation.
• In many cases, I have changed the title of the bookmark to be something more meaningful to me than the page’s url. My method preserves these changes, whereas recreating the bookmark in the way you suggest would need me to rewrite it.

I have now corrected the second of my smaller projects. I now have two more projects to go. These are both much larger, one with 321 containers, the second with 885 (over 2,500 bookmarks in total). I have already lost a lot of writing time tackling this problem, and I really do not want to lose the days it would take to manually update these projects. As you rightly say, what counts is the precious writing time.

I still hope that L&L will do the right thing and issue a fix.

[simeva]

@AmberV

do bear in mind that it is way more complicated for us to create a tool to fix it than for an individual. A script that fixes one project can be very specific, very simple. We have to build a whole kitchen when we build something, because it needs to work flawlessly across hypothetically tens of thousands of projects, from all eras of usage. Then there is the packaging of it, making a GUI since most people would throw their hands up and yell at us if we made a command-line tool. Then there is testing, which takes a lot of time. Do we need legal to look at it and make sure our EULA covers us from someone blowing up both the backup and the new copy in the unfortunate case that happens? In short we might do this, but don’t expect something tomorrow.

@AmberV - Indeed I had not considered these issues.

It surprises me that no-one else has joined this discussion. Tens (hundreds?) of thousands (millions???) of users must have updated projects to Scrivener 3. Every single one of them will have lost bookmarks.

Unless I am the only one that uses this functionality? But perhaps that’s good news, for then there’s only a need for a solution that fixes my two projects. Given the amount of the data that I have lost from this bug, I’d hope you might still feel an obligation to help rather than putting me through days of mind-numbing copy-and-paste in an XML editor.

I hope my analysis above is helpful, and that when the developers get back from leave they will be able to spend a little time on this.

[AmberV]

It surprises me that no-one else has joined this discussion. Tens (hundreds?) of thousands (millions???) of users must have updated projects to Scrivener 3. Every single one of them will have lost bookmarks.

A lot of people don’t really dig into the software, and that can’t be underestimated when it comes to things that are effectively buried in secondary tabs of sidebars that may not be immediately obvious as to their usage, nor would there ever be any strong impetus to figure out what it is—like say the Compiler has. References, in v1, were pretty simple and extremely disconnected from everything—in fact I often didn’t even use them and just used lists of links in Document Notes instead, because that was easier to work with and had an export path. They are a far cry from the more integrated and useful tool that Bookmarks are in v3.

So yeah, it doesn’t surprise me that not many people have run into this.

[narrsd]

@simeva

the Ishiguro interview, which I do like - it’s candid, unpretentious, direct. It’s inspired me to start When We Were Orphans, which I have not read.

Glad you liked it, as it’s fun to share, and I found it quite inspiring as well – towards writing. For some reason I’ve liked his second novel that he mentions quite well, read it again recently. Maybe because it is subtle about things not in Europe, and for some life I had in Asia.

That’s An Artist of the Floating World, and besides being clear and humane with his interest in what is not, always, it has a lot of presence over time, just so in an artist’s world.

@simeva

I prefer using the XML editors to move these bookmarks because:

It’s all great, Simeva, that works for you , and appreciated.

@simeva

I still hope that L&L will do the right thing and issue a fix.

Yes, of course. I’m looking at your most recent conversation with @AmberV , as how you stated it is for me the best and easiest answer as I’ve thought as well.

Good hunting, on your conversions, and hope it gets easier, with a fit to days that keeps a good amount of your writing time.

I’m sure it will be an immense satisfaction, and well earned, as you close on this; and then it’s always more fun agan to do with what we’ve made, am sure you know also.

Best,
Clive

[narrsd]

@simeva, a couple of corrections to above, if you are reading by email so would miss result.

I really have a very bad habit of editing after pushing Reply or Send anywhere…and even with this nice forum that shows how things are going to turn out, if you look at them beforehand…

[narrsd]

@AmberV

do bear in mind that it is way more complicated for us to create a tool to fix it than for an individual.

Hi @AmberV , I think the suggestion is that a release of Scrivener Windows that actually contains the code to bring over external Bookmarks when converting a version 1 project would be the suitable and easiest answer.

From a coding view, all the tools and structures are right there to use, and it’s not a complex process if you just do it directly at that time, in the flow of moving other things. Enormously simpler and safer than a transferring corrector…at least as I may see it.

You’re surely right about everyone not knowing or using all the possible tools in Scrivener – it’s how I got into recent months’ discussions here, to learn for further ones I can make good use of now.

This Bookmarks thing has to be just an oversight, as happens in big projects, and I guess I’ve seen enough of such out there ‘in the world’ of persons getting such surprises, to simply feel the good judgements of getting a fix on it, as originally happens.

And then it’s your guys’ call, for sure and always.

[AmberV]

I really don’t think going down the path of adding bug fixing tools into the core software is a good idea. While it does mean less “packaging” overhead, it sets a bad design precedent in my opinion.

Besides, it’s not like the libraries for reading .scrivx files are unavailable to us outside of Scrivener. The software is coded in a modular way, and that means a tool could be made from those libraries that can read v1 and v3 formats and do stuff with them.

[narrsd]

@AmberV

I really don’t think going down the path of adding bug fixing tools into the core software is a good idea. While it does mean less “packaging” overhead, it sets a bad design precedent in my opinion.

Amber, I think we are just seeing this differently.

For me it’s not ‘bug fixing tools’. It’s correcting basic operation, so that when you let Scrivener auto-transfer a project from version 1 to version 3, it includes the external (web link and file link) Bookmarks along with everything else, as would be expected, not so?

That’s all, and again, it’s all up to you guys, isn’t it.

[AmberV]

We might be talking about two different things here. The core project upgrading process will be fixed, and already is in internal builds. I can no longer reproduce this error when upgrading old projects fresh. This bug will not be a problem after the next update.

What I’m talking about is a tool that goes back and retroactively fixes a project that wasn’t properly upgraded in the past, that is too far edited to consider going back to the old v1 backup and starting from scratch. That’s not a part of any core process in Scrivener, and would absolutely necessitate a whole new interface being built into the software, with some kind of “Import Missing References from a v1 Project…” menu command, and documentation, and so forth.

[JimRac]

@AmberV

The core project upgrading process will be fixed, and already is in internal builds. I can no longer reproduce this error when upgrading old projects fresh. This bug will not be a problem after the next update.

You’re saying that production Windows Scrivener v3.1.1.0 still has this upgrade bug where v1 bookmarks are lost?

[AmberV]

Sadly yes. The focus has been on Scapple for about six months. I’m not happy about it either, but hopefully things will get better soon.

[narrsd]

Ok! We’re on the same page, then, and it’s great to hear a) that there’s a bump in builds coming, and b) that this will be in it and already verified by you.

While you were replying (and I happened to still be on the page so as to see the indicator flashing, relates to other discussion on cloud transfers ), another thought returned, which I hope it will be helpful (truly) to mention.

It’s that the ‘world’, which I still deal in a bit for several reasons, has moved on from thunderous and agonizing ’ full releases’, to a trickling system that would be as comfortable as the management that is a shadow over any work were, which it’s clear you from Keith on down take every effort to avoid in the L&L including Scrivener efforts.

I’m in very full sympathy with you, and any amount of my consulting work was untangling such, though we needn’t go into that even if there are good stories, not in any framework for fiction I am trying to be writing…

It’s just that it does often seem that it would relieve the pressures that show in support conversations, if you guys could release one small but useful package of improvements every now and then, thus in a relaxed way.

This would mean fixes and features could come more comfortably often. It seems this would be pleasant for everyone, including yourselves, in a way first of all.

Once again, I’m not talking about the machinery for still too industrial processes, but a genial situation where when you’ve tested the fruit of something like this improvement we’ve been discussing, it could just go out then, update for everyone, with life then going on a little better.

I imagine it, and leave the subject, as ‘that simple’. Could be, as always, that the story would be a little different, but that’s some part of why we write them, no secret

Best, and a nice Spanish night I hope it is, imagining that also, from memories…

Clive

[AmberV]

@narrsd: It’s just that it does often seem that it would relieve the pressures that show in support conversations, if you guys could release one small but useful package of improvements every now and then, thus in a relaxed way.

I do agree, and hopefully we’ll see improvements in that area in the future. There is of course a balance—you don’t want to flood people with updates, particularly if you have an audience that includes a lot of people who are very nervous around such things, but waiting an entire year or more between updates is too much, especially when really important stuff like this starts to stack up.

Best, and a nice Spanish night I hope it is, imagining that also, from memories…

I could do with it being about 10° cooler, which would make it a breezy 30°. But otherwise, yes. Thank goodness for what remains of the siesta culture sometimes.

[JimRac]

@AmberV

Sadly yes. The focus has been on Scapple for about six months. I’m not happy about it either, but hopefully things will get better soon.

Wow. That’s disheartening. And kinda lame.

[simeva]

Here is another thought for anyone else facing this. The second project I fixed was a regular book project, with a fairly clear distinction between the Research and Draft folders. Almost all the lost bookmarks were in the Research folder, where the text had not changed, and almost all the new writing was in the Draft folder. I was therefore able to recover most of the missing bookmarks while keeping the new writing like this:

• open the project’s final 1.9 back-up, the one with the bookmarks, (OLD) in Scrivener 1.9
• open the current version (NEW) with missing bookmarks in Scrivener 3
• arrange them side by side on the screen
• inside the NEW version, make a copy of the entire research folder contents in a new folder called ResearchBak (just to be on the safe side)
• drag and drop the entire contents of the OLD research folder into the NEW project.

@Amber: way back, you said:

@AmberV

The more labour-intensive part will be finding the individual binder items since the internal IDs will have changed.

Are you sure? My observation was that the internal UUID attribute for each container was not changed by the upgrade. I used this ID to search for containers, and in each case found the right one.

This would be a critical factor for a software solution.

@JimRac

Wow. That’s disheartening. And kinda lame.

Exactly. Every user who has upgraded a project from 1.9 to 3 has lost every one of their external bookmarks. Anyone who upgrades a project today will still lose their bookmarks.

[simeva]

@simeva

drag and drop the entire contents of the OLD research folder into the NEW project.

It’s just occured to me… is this an okay thing to do? Can you safely drag and drop parts of S1.9 projects into S3 projects?