Using Scrivener to edit notes spread across folders

Scrivener Wish List
1

Hi, this is my 2nd day evaluation. I read the tutorial and like your program very much, though may not use it, since it is different to my workflow. But I think we all know, that this is a personal preference and does not mean, your program is bad. Quite contrary! It’s great for what it does, except a few points I came across right when trying to import one of my projects (which are mixtures of essays and code or Celtx script projects (my hobby))

a) I could not import a text/plain if it did have a uncommon suffix (*.todo). When I changed the suffix into *.txt it would import flawlessly.

b) not being able to im/export to iWork pages and ODT is a major turnoff. Apple’s ODT support is too basic to be considered serious.

c) what about giving the user a little bit more freedom when it comes to project structure? I would love to have a project being in sync with the folder structure on my disk. Nothing in Scriveners interface would change. I could, however, define a target folder on my filesystem for every Scrivener folder, and if I create a new sub-folder, it gets mirrored to my disk. So I would end up with something like this on my harddisk:

/Users/ user/ Desktop/ Projects/ -> project A/ -> drafts/ -> research/ (these are synced with Scrivener, Synopsis could go into the Spotlight comment of files and folders and filenames of files could maybe be different, but not by default, since it complicates things) website/ code/ localtest/ production/ (ftpfs/sshfs link) contacts (Spotlight SmartFolder) etc.

With updates (since such thing is a lot of work) tags and labels like To Do could be mirrored to the outside of Scrivener as well. For example as OpenMeta tags, allowing to create SmartFolders with ToDo items for a certain project (which must not all be Scrivener todos, btw., hence this idea)

I would like to see a step away from monolithic applications, that do it all alone and have more possibilities to access my data from outside of the box more easily, in order to combine my projects with other productivity applications.

2

Thanks for trying Scriv.

Sadly, suffix is still the best way of determining file type (creator codes are not consistent enough) so Scrivener has to rely on them to determine whether a file is of a type it can import or not.

Regarding .pages, you need to talk to Apple about that. Apple do not provide importers/exporters for the .pages format to developers, and have not made the .pages format public (beyond a developer page giving a subset of the details of the Pages 1.0 format). So Apple do not make it possible for developers to support the .pages format. If Pages supported RTf well enough, it wouldn’t be a problem. I agree that Apple’s default .odt import/export isn’t great either, but as a single developer I have to rely on it. I have done a lot of work on Scrivener’s RTF import/export, and that is by far the best format to use. Sadly neither Pages nor OpenOffice support RTF very well, but I can’t really be held accountable for the limitations of other programs. Pages and OpenOffice are very limited when it comes to RTF, and Scrivener is limited when it comes to .doc, .docx and .odt - sadly that’s the way it is with the exchange of formats, it’s never great.

You are free to structure a project as you like - Scrivener never tries to govern or limit the structure…

…But on the other hand, that’s not possible

Actually, lots would have to change. File systems have limitations on file names, and you’d never be able to have different documents with the same title in the same folder for a start.

But also, Scrivener projects are supposed to store the information you access - they are supposed to be project depositories.

All the best,
Keith

3

A lot of flexibility would be lost by switching to what would essentially be a file browser. The identical name issue already mentioned is one such important thing (and tangent to that you can name things however you like in Scrivener; if it were tied to the file system there would be annoying limitations, like no colons), another is that folders are really more a formality in Scrivener. They have some important differences, but at the data-storage level, are identical to documents. You can store text in them just as easily as in a document, and likewise documents can store other documents beneath themselves like a folder. Indeed you can even switch back and forth between the types by right-clicking on the item in the Binder. There would be no elegant way to express this flexibility using file system conventions.

Further, something that wouldn’t be immediately clear is that Scrivener basically hides the fact that a “document” or “folder” is really a conglomeration of several files. This is all basically hidden from the user, but notes, synopsis, and further features in the future are all tied together with a single click in Scrivener. To do this in an open file system would create quite a bit more “mess” than you’d expect.

Lastly, Scrivener does optimisation which keeps projects responsive and searches quick. Part of this optimisation relies upon Scrivener being the only application which touches its files. To keep things quick, Scrivener would have to re-analyse the entire project whenever it was opened, increasing load times. For small projects that might be trivial, but some people have fairly huge projects, and synchronising strings every time they opened the project would be very taxing on one’s patience.

4

Thanks for the generous 30 running days method of evaluation :slight_smile:

Why not let the user define a bunch of extensions, which get associated with filetypes Scrivener understands?

I know :wink: I have the docs here. And a project lying around for months, that should have become an iWork XML to XHTML XSLT. I lost fun with it. Still tell myself every day to dig deeper into it. I did not know, that there is no other solution, however.

I absolutley understand. When I saw the file-formats you support, I immediatly knew, you’re a one man shop. :slight_smile: It wasn’t meant as critic. I just needed a forum to talk about my feelings :mrgreen:

Too bad… :cry:

Maybe the user would accept this as a tradeoff for that mode? I certainly would.

Yes, but for this, they are too limited. The problem behind this is really being big, these days. It is a generic project-management problem. A Scrivener project can not be the project itself, since it does not contain the final outcome, the export. Nor does it offer a quick way to access other data sources, that are not being supported.
I know, that not anybody is doing it like me, but that’s the trck behind my thought. Everybody has another way to organize the workflow. When I create a new project I have a fully preset project template which is a folder with certain sub-folders. It contains even SmartFolders to search for contacts, emails, shedules and meetings, tagged with OpenMeta (or other Spotlight data). Along with this template I have folders called ‘Code’, ‘Media’, ‘Drafts’, ‘Outgoing’.

The project I am working on right now is the one I have imported into Scrivener. It consists of an essay about information management. I do a lot of experiments with diagrams as well as with mind-maps, in order to see more about what I am doing. Another thing I am doing is trying to develop an example implementation (a framework), that tries to solve the problem and prove the hypothesis I show in the essay. As you can see, Scrivener is, by concept, very well suited for it, since it is an essay development work-horse. On the other side, the project is much more complex, than Scrivener can take, atm. If Scrivener had a file-mode (in addition, not replacing what it curently does), I could use it as a frontend to my project, having it, at least, tag and annotate files, that I can not import directly, or have it import PNG exports of the diagrams I am doing. The code would still be developed outside of Scrivener, but, hey, what about some sourcecode import, with colorization (there is quite some open-source libraries for this)? This would make for great code examples within the essay.

Of course, I am aware, that Scrivener is a tool for literature at first. But this is also the essay I am writing. With the difference, that it is not fictional, but of scientific/philosophical nature, meaning, that I need to have some illustrations of code.

The user could chose, which mode (s)he desires. Scrivener as a frontend to an outside structure, still retaining (most of) its functionality, making it a way to view data through special, enhanced glasses, or the classical way.

I hope you don’t mind me going into this so lengthy, it’s just, that the work i am currently doing circulates about these topics :wink:

Thanks for your time and kind support.

Andreas

5

AmberV, by no means I would want to see anything being replaced. Just added.

As for the identical name issue and many of the other things you mention, there sure would be tradeoffs. But couldn’t it be, that someone like me likes Scrivener, though still does not need all the functionality?

Sure, additional data management could slow things down. But the clutter you talk about could be solved by either storing a Scrivener bundle with the additional data in the project’s folder (maybe as an invisible file) or by keeping some stuff in an sqlite db. Both are less elegant, as a fully contained project in a private bundle, of course. A lot can be done by using Spotlight data, especially OpenMeta.

Of course, Scrivener has not been created for the work I am doing. It is clear, that it has been developed for Scriptwriting and literary work, such as novels and essays. And I think Scrivener solves this problem very (!) elegantly. I really like it. And because I like it so much I would have been glad to be able to use it for writing work, that has some variation, some accent, so to say, to what Scrivener can do already.

Thanks for your thoughts and input.

Andreas

6

Hi,

Not a bad idea for the future, maybe.

Hmm, funny - I think Scrivener supports a lot more formats than many team-developed products, but hey. :slight_smile:

As I say, this just isn’t feasible. It would require Scrivener essentially being a different program entirely - if this is what you are after, something like DevonThink or EagleFiler may work better for you, I’m not sure.

I’m not sure how. Most users find the ability to import research and view it alongside text a “godsend” (I’m quoting form a Twitter comment I received half an hour ago there :slight_smile: ). And that is what I implemented this feature for - so that I could view images alongside text where I needed, or a PDF file in a split view. It’s not intended as a repository for vast amounts of research, but a place where you can keep what you need to refer to for the current project. But of course, no program can suit every single user’s every needs.

I don’t think that makes any sense. That’s the same as saying that an iMovie or Final Cut project isn’t the project itself because it doesn’t include the final outcome, the movie. Or that an InDesign project isn’t actually the project because it isn’t a PDF.

Everything said above about the problems of using the file system to replace the current system also applies to having it as an addition. Synopses, notes etc being separate files, Scrivener needing to know about files being edited externally, losing formatting from file formats not fully supported etc - all would still be issues.

You can already import image files.

Scrivener isn’t a code editor, so this is well outside of its scope. As soon as you start saying, “Why not…, and what about…, and why not…”, the list never ends - and it will be a different list for every user.

I hope that makes sense. I understand why you want this, but I’m afraid it’s not something that’s going to be added - it’s just impossible to suit every single workflow, as I’m sure you can understand.

Thanks and all the best,
Keith

7

I just want to say that the problem of a monolithic package or using the file system is a problem that I am having with other programs. Two other programs that I use, DevonThink and Sente, allow one to either use the Mac’s folder structure or to save everything in a bundle. There are real trade-offs in each, and each program has some design inconsistencies when you move from one work-flow to the other (and sometimes you can’t move from one to the other, I think).

I am becoming enamored with OpenMeta, and it would be lovely if Scrivener could jump on this bandwagon, but I think it would mean abandoning the package structure. Apple really did create some very basic usability issues when they introduced the idea of packages.

8

Given that the internal files are not meant to be messed with by other applications, I don’t think the bundle format is a bad idea. If you put everything out in a regular folder structure, that just increases the temptation to start editing files using whatever program you want, and that can cause serious issues if mismanaged. The bundle is a good trade-off between protection and accessibility if necessary. The trade-off here is that if Scrivener has total control over the project files. Everything that I’ve already said above: duplicate file naming issues; it can remain optimised with them and provide features to them that most other applications on the Mac cannot serve—and this will be even more the case moving onward. If the files were public, and you opened them in TextEdit, you could potentially lose information in the file that TextEdit (and hundreds of other Mac programs based on it) would be unable to deal with. If anything could edit the internal files, Scrivener would have to re-optimise its search indexing every time you opened the project. With large projects, this could cause project open durations to sky rocket. Given all of this, I think hiding things away is a very good idea.

OpenMeta: what would be the purpose of applying this system to the files within the bundle at all, if the files within the bundle are not meant to be system public? What would make more sense to me would be to apply OpenMeta extensions to the files that are produced by the Export feature, which is all about sharing the Binder with the world. Since the Binder Export feature does not use bundles, there is no need to abandon the format to apply OpenMeta to the files which are meant to be system public.

9

I can imagine scenarios where I would want to tag parts of a Scrivener “file”. I have tagged references according to the species that are discussed (I have to use tags because a textual search might only show the scientific name as used in the document, not what is currently being used). I would like the Scrivener section to also share that tag. That being said, it is a small request, and I can live without it (in fact, I can live without OpenMeta tags, and have done so for half a century).

Note–associating a tag with a part of a file is within the realm of the possible. I don’t understand the details, but the OpenMeta documentation suggests that it is possible to tag parts using AppleScript. For instance, Mail (which I don’t use) is reputed to allow the tagging of individual emails. Emails are, of course, read only.

Your idea of putting the tags in exported files is interesting. One could use tags internally to tag parts of a document in Scrivener (Scrivener would be smart enough to allow one to draw from the external tagging pool). And then when exported, all the tags in the parts would be associated with the output.

That would be cool. I can’t imagine it would make the grade to be included in 2.0 though. Maybe in 2.1?

10

Could you describe more precisely what you are looking for? I think I might be misunderstanding the usage of terms here. When you say “file”, are you meaning a document in the Binder? It sounds like that is what you are doing—and assigning keywords to documents in order to highlight themes that are not actually present in the text is an excellent usage for keywords. But you later go on to say you’d like to apply keywords to a Scrivener “section”. I’m not sure what you mean by that. If we are crossing words here, and you are not aware of it: if you open up the Inspector ([b]Opt-Cmd-I[/b]) and click the key icon in the bottom, you can enter keywords (or “tags” if you prefer to call them that, but Scrivener uses “keywords” throughout the interface so I’ll stick with that) into this list. When you do so, these get added to a central list in the project which can be accessed with [b]Shift-Opt-Cmd-H[/b]. From there you can just drag existing keywords onto documents, organise them into hierarchies, and do quick project-wide searches for them.

Sorry if you already knew all of that, but when you said you’d like to apply them to “sections”, I figured I ought to cover all of the bases.

I’m not aware of the ability to add meta-data to only parts of files (again, we might be using different meanings of the word “file” here, I use it to mean any entry in the filesystem that has associated with it a data stream (which might be empty, or in some exotic cases can be the raw input data, like from your mouse)), but it sounds interesting. It’s probably not something you’ll see in Scrivener though because by nature it approaches documents by cutting them up into manageable pieces. You already can add meta-data to small parts of a manuscript because the manuscript is, in most cases, cut up into discrete chunks. If a single binder document is so long that it requires a different approach to meta-data in different parts of it—that’s probably an indicator that it should be split up (at least in common “best practices” usage of Scrivener—there are always outliers).

But, if your example of this is simply applying a keyword to an e-mail, I’m not sure if that should be classified as sub-document meta-data because technically each e-mail is a separate file under the system that Mail uses. It doesn’t use the old mailbox file format where “folders” are really long files with hundreds of e-mails in them. Each individual e-mail is its own .emlx file.

Mail could be accomplishing this in a number of different ways. While e-mails are certainly read-only from the user end, what the application can do with the file is definitely not read-only. You could try it yourself if you ever synced using Mail. Open up an .emlx file in a plain-text editor and you’ll see it is just an ordinary text file. You could change what is inside it just as easily as editing any other file. E-mails being read-only is kind of a collaborative user client agreement to not allow it. :slight_smile:

Secondly, OpenMeta is stored in a special extended attribute portion of the file, and once you get down to this lower level of things, files start to look less like single files. It’s really almost more like two files (okay, three if it has a resource fork) that look like one file, kind of like a Bundle :slight_smile:, but so tightly bound that you’ll never see that unless you are using some seriously low-level tools. Thus, even an actual read-only file could have its extended attributes modified. To my knowledge there is no lock on that.

Aside: This is why I’m not so sure about sub-file meta-data. OpenMeta operates in a space that is attached to a discrete file entry in your computer. While it is conceivable that it could use byte-length addressing system to “fake it”, that would be very fragile. Just add a single letter to the file and the byte-length changes—skewing all of the offsets. Without an anchor in the data stream, it would be extremely difficult to pull this off without errors.

Again, I don’t think “parts of a document” is so crucial in Scrivener since its fundamental philosophy is breaking down a document into thematic pieces, but I see no reason why Scrivener’s document keywords and some of its other meta-data might make an interesting transition to OpenMeta upon export. I wonder about its practicality though. OpenMeta makes all kinds of sense in a library and catalogue context, but Scrivener’s export is more often used for backup, collaboration, and multi-platform integration. On the latter, OM cannot even exist. Collaboration could be interesting, but I seriously doubt many would find a use for it. OM is more about finding resources without worrying about where they are. If you get a zip file from a colleague with a bunch of Scrivener files, you’d have to be pretty absent minded to not know where they just unzipped. Backup is probably the most logical, so long as you don’t zip them. I’d bet most people just backup the whole project though. You lose a lot of structural project work with Export. That’s more for very-long-term “I might not have Scrivener in 30 years” type backups. Do these files need to be coming up in Spotlight searches? Maybe…

This is just my opinion, but I think having some vast system-collective keyword pool cluttering up the project pool would be kind of annoying. One of the first things I did when I got Aperture was delete all five million keywords from its HUD because it was taking way too long to find anything appropriate. I’d much rather just type it in at that point. It’s just a word. Perhaps that is a bad analogy, but I rather like how each project has its own keyword collections. Consider your project—would you want all of those Latin words cluttering up a biography you decide to write in two years?

11

AmberV

You posted a long letter, and I have been traveling so sorry for the delay in responding.

I probably was unclear because “document” on the Mac and “document” in Scrivener are different things. What Scrivener calls a project, most people refer to as a document on the Mac. I’ll try to use Scrivener terminology.

What I was trying to express was the desire to be able to tag a document in Scrivener, and have it show up in a tagging program (like Tags, Yep, Leap, etc). So, for instance, I have a bunch of pdfs that are tagged (not yet tagged using OpenMeta, but that may be coming) with “Physeter macrocephalus” which is the Latin for Sperm Whale. Many of these documents actually don’t contain the phrase “Physeter macrocephalus” in their content, because for a long time most zoologists referred to the Sperm Whale as “Physeter catodon” (long story as to why that is). Tagging allows me to find these pdfs easily.

I was thinking it would be useful to also be able to tag a document in Scrivener with the same tag, and have it show up in, say, Spotlight searches, or in a tagging programs.

However, this is (a) a low priority for me, (b) just a suggestion, and © you might well be right when you say “I think having some vast system-collective keyword pool cluttering up the project pool would be kind of annoying”. It might well be annoying.

I am only beginning to use OpenMeta tagging, and starting with the easy stuff and will graduate to the more complicated later if I need to.

A