Large-Scale Info Managers and Scriv

Latte Other Software & Development
1

I just discovered the humorously-named circus ponies notebook, and I was wondering if any of y’all could share your experiences with using it. I’m kind of desperately in need of a global information junk drawer/collection software, and right now I’m torn amongst DevonThink Pro, Journler (which is just unbelievably cheap), or whatever else is out there. Since Scrivener is more or less my favorite program ever, tight integration with Scriv is a MUST – in fact, that’s one of the big hurdles towards DTP for me.

As it stands, the workflow looks like this:

Universe of Information -> [DTP/Journler] -> Relevant Info into Scriv’s Research Folder -> Write!

As a second question – in looking around the forums, I’ve basically deduced that, in order to realize the true power of DTP, one must put a truly mindboggling amount of information into it. Where’s that threshold? I mean, I have a lot of stuff, but not the gigs and gigs of text and clippings that, say, Merlin Mann puts into it.

Thanks so much for the responses,

Brian

2

Brian

I’m not an expert on this subject - there are others on this forum who are - but I am engaged on a similar quest to yours.

I found this blog interesting: http://www.davidhewson.com/content/view/26/51/. It was written by another forum member, bodsham, a novelist.

I’m still in the extended trial period of DTP. I’ve imported about 300 documents into it. I don’t know whether I’ve passed the magic threshold because I haven’t attempted to use the Auto-Classify function yet, but I have found the extensive Search function very effective. However as you say, it doesn’t integrate with Scrivener perfectly; also, price is a factor.

Eaglefiler is another option; I haven’t tested how well it works with Scrivener, but I suspect it may be slightly better than DTP in this respect because unlike DTP it stores files in their native formats. But it lacks DTP’s so-called AI. Swings and roundabouts!

However, I think there’s a danger with all these pieces of software that you spend more time playing with them than using them. They become toys rather than tools (as bodsham hints in the title of his blog). I think in the end, unless there are overwhelming reasons to the contrary, you probably just have to plump for one with a reasonable reputation, and make it work for you. :slight_smile:

H

Edit: There is, apparently, a name for the “illness” of endlessly trialling software of this type: CRIMP, which stands for “compulsive-reactive information management purchasing”. Seems to infect males more than females, probably for Nick Hornby-ish reasons. Not that I’m suggesting anybody here is a victim. :wink:

3

I’ve tried all sorts of these info-management programs, and to be honest, I find that they’re usually worse than useless for me - I spend more time trying to get everything set up in them than I do actually using the stuff I’m so carefully filing. Without fail I always go right back to using the Finder and - gasp! - a filing cabinet and 3-ring binders. Maybe I just don’t have that much info, or my brain just doesn’t work that way. I don’t know.

4

I use Circus Ponies for easily categorizable information, such as registration codes, frequent flyer numbers, stuff like that. It works well for that purpose, but I wouldn’t recommend it for large scale information management. The notebook metaphor makes it too difficult to find and rearrange things on a large scale. (Either large items or lots of them.)

I’ve found DTP to be indispensable. My primary database is about 1.75 million words; I don’t know if that is huge by your standards or not. It was useful long before it reached that point, though. If you have too many items to find by eyeball searching, then DTP can help you. If you do a lot of web research, the DevonAgent/DTP package is well worth the investment, as DA is designed for dumping data into DTP.

I’m not sure what you mean about Scrivener integration, as I spent Monday dragging PDFs from DTP to Scrivener with no problems at all. What are you trying to do?

Katherine

5

I second the EagleFiler suggestion. It has a much less radical interface/storage philosophy than DT/P, so you can get into it much faster. The tag+folder system also works right from the start, whether you have 7 or 10,000 documents in it. And anything that doesn’t use a database to store what are essentially just files gets a plus in my book. This was the single biggest strike against Yojimbo for me. It has a much more “slick” interface than EagleFiler, but stores everything in a database.

And on the matter of integration with Scrivener, raw files will win every time, since Scrivener supports so many file types on import. Importing an EF folder into Scrivener is no more difficult than dragging a folder in the Finder into Scrivener’s Binder.

6

After trying them all, my suggestions for information managers are now limited to Dt Pro, Eaglefiler and Yojimbo.

I think it depends on your needs. If they are more modest, Eaglefiler and Yojimbo are worth a try (I agree with the Eaglefiler suggestion). I don’t trust Journler enough yet to give it my precious research data and I’m not sure how easy it would be to manage a really high-level of information with lots of files within it. Circus Ponies was just too cumbersome for info management. Might work fine as a note-taker, but even there, it just didn’t work for me. I gave it a good go at one time because I thought it was a cool program. Moris is way too limited in what I can handle.

I use DTP simply because it handles large-scale information better than the others I’ve tried and I have a LOT of research data. I actually don’t use DT’s AI features all that much to be honest. Mostly I just trust it because it is extremely stable and has powerful enough search features for me to find what I need. In the two odd years I’ve been using DT, I believe has crashed only once and I never lost any data at all. It’s fast and stable and can take what I give it. It is very limited in other ways regarding the UI and lack of smart folders, tags, etc., and can be a challenge to get your mind around how to use it.

Sorry I can’t be clearer, but I think you’ll need to give them a try yourself to see how they work for you.

Alexandria

7

An intermediate solution is DevonNote, which provides much of the functionality of DT/DTP at lower cost. It worked fine for me for a couple of years, before I decided to just manage all my info in Scrivener. Very stable – not bad for a $10 (on sale) app!

Still, if I were in the market now, I’d take a long look at EagleFiler, for the reasons enumerated above. A bit spendy, but I like its style.

For my needs, Scrivener seems to be plenty, which is a joy for someone who strives to reduce clutter, including app overload.

8

As a long-time user of DevonThink products, I must vote to wait for v2.x of DT or DTPro. The UI is confusing and it almost integrates with the OS, which can be as frustrating as non-integration.

Journler’s developer is working on a product called “Lex”. It promises to be a decent info manager, similar to Journler but built to manage info. Journler isn’t quite an info manager, in my view.

I want to like Eaglefiler more than Yojimbo but it is too slow, compared to Yojimbo, on both my Intel Macs. It’s not too slow to be usable, it’s just that I’m used to Yojimbo’s near instantaneous response and I also like it’s tagging and bookmarklet- web capture system. I haven’t really given EF a go for a few months. Perhaps I should try it again.

The Finder or PathFinder are nearly very good info managers if used in conjunction with file tagging or something like tagbot. Spotlight’s UI, search terms and search results are the weak link in that chain for me. there, DT shines.

YMMV

9

But how much stuff do you want to archive? Some hundreds of articles and say a thousand loose annotations? Then you can take almost any recent application of this kind, including Scrivener and the free Journler.

Thousands of books and articles, and countless annotations? Then I would go for DTP. To be honest, I don’t really like this program, which as alexwein and bluloo have pointed out has some clear deficiencies: confusing UI, lack of a tags / keywords feature (incredible for a similar application!) etc.

But DTP can effectively handle huge amounts of data, and it is indeed very reliable: I have been working with DT for some two years now, and never have experienced a crash.

Moreover, version 2 of DTP, which has been announced a long time ago, should become a really big update. We’ll see.

10

I am attorney using Journler (for document keeping) and Scrivener (for writing). Both allow you to organize, store and find documents of various types. Journler is better suited for a paperless filing system for all kinds of notes, documents, www sites and many projects. Scrivener is better for writing and keeping one project’s drafts and research materials together in a single “project“ or folder.

Scrivener’s advantages are well documented here and in www reviews. It is quick and easy to learn and implement in my work flow. It’s amazing how much easier drafting agreements can be when prior drafts, emails and other materials are right at hand in the binder. In fact, Scrivener, with its index notes and split views of multiple documents, may the best writing tool I have ever used.

I have tens of thousands of documents and emails that I keep and use. Before Journler, I kept projects, notes, forms, research materials, pdf’s, www bookmarks, etc scattered in various folders. Different word processors, mail programs, etc were required to read them. Spotlight was insufficient to find them, and I used SpotLaser as a Spotlight front end to search. Although my file naming practices and client folders were consistent, it was a filing system slowly breaking down.

I needed help. I needed a simple filing system that offered more than the Finder.

I tried Journler’s competitors such as Devonthink, SOHO Notes, Eaglefiler, Yojimbo, Mori, etc. Each has different capabilities and features, but I found that Journler’s combination of nested folders, smart folders, ranked search, auto-tagging (with smart folders), and finder friendly database most useful for me. Currently I am copying documents into Journler and am not relying upon aliases to external files which might be broken. My Journler database has about 3000 files, is about 500mb - and is growing.

Capability for handling large numbers of documents and speed should always be two important priorities for Journler. Journler is quickly evolving beyond its journaling origins. According to user comments on the www, Devonthink and Eaglefiler may be able to efficiently handle significantly larger numbers of documents (gigabytes of data) than Journler. I don’t know yet how large a database Journler can handle. And Devonthink and Eaglefiler may be faster in importing and indexing documents. I find that importing more than 30 documents at a time bogs Journler down with the spinning beachball, or it crashes.

Why pick Journler then? I did not find Devonthink’s auto-indexing AI and tagging useful. I could not tell where my documents went. I will file my documents - I did not want Devonthink to guess where they belonged. Eaglefiler is in the early phases of development and does not allow you to create smart folders. SOHO Notes did not appear to have the support of its users (a bad sign when they took the user forums down), and it left program parts scattered throughout my system. Yojimbo has no nested folders. And Mori just never grabbed my attention.

A thoughtful review of Journler and these other programs is at:

musingsfrommars.org/2007/03/ … -os-x.html

However, all have been updated since this review.

Importantly for me, Journler is simple and quick to learn. It has an active and supporting user community and development. To utilize Journler’s auto-tagging (which I found to make the most sense of all of the programs), all of my folders are organized as smart folders and nested smart folders. I have Notes, Client and Forms smart folders with nested category or project smart folders within them. Drag and drop a file or files to the desired smart folder (or nested smart folder) and tagging automatically happens. Like some of the other programs, I like the integrated pdf, www, and document readers and Safari-like tabs.

I don’t do much Wiki linking or resource linking within Journler, although the capability is there. I rely upon the smart folders and ranked search to find materials that may relate to each other.

And I look forward to working with Lex more. I hope that it might work with search phrases - and not just single words. Like Google. Thousands of documents create thousands of word links - when a phrase might make the search process faster and simpler.

One feature I would add to Journler now is a preference option for email dates. Instead of the current date, I would like imported emails to use the date that they were sent or received.

I hope this is useful to those of you considering and using Journler and Scrivener together. Journler is a wonderful program and complements Scrivener well.

11

@erooke:

Thanks for that! I have been using Notational Velocity for various simple notes but was looking for something a little more capable for other odds and ends. Yojimbo is good for some things but Journler looks like a winner. Trying it out now.

12

I used Circus Ponies Notebook before I discovered DevonThinkPro. Notebook is brilliant for clipping from the web and making your own notes and outlines. I don’t use it now that I have DTPro.

I found DTPro useful from the get-go. I began with a base of just over a hundred documents.

I haven’t had any problems integrating DTPro and Scrivener. I love that I can drag documents from DTPro into Scrivener. Scrivener seems to handle conversion automatically.

13

erooke, that was very helpful. You have used Journler more than I have. In a much earlier incarnation, I used Journler, but it was very, very glitchy so that’s why I don’t trust it. But that was early in its development process and it has definitely grown since then. I would like to be able to trust it for the reasons you mentioned.

I’d LOVE to be able to do what Brett does and just use Scr. for everything, but I find that materials get spread across projects and (1) they are harder to find and (2) I have to duplicate research material.

But your evaluation of Journler was helpful. Another problem I would have in changing is wiki links. Dt pro allows me to use file names and aliases for wiki links, which is the way I have it all set up. I’d have to change file names for all my wiki linked files.

Oh, well. I wish I loved DT as much as I love Scr. and then I’d not even consider switching programs!

Alexandria

14

I use CP’s Notebook for all the info bits I like to collect, primarily because its clipping service makes it a breeze to highlight, right click, scroll to the name of the notebook I want and click. The information is saved. I even have a notebook set up for Scrivener so I can collect all of Amber’s shortcut posts. :smiley:

It’s a very useful program, imo, if it does what you want. I’m sure there are many more features that I don’t use, but it’s grand for what I am using it for.

I’ve also been able to set up a date-due listing of research books from the library, which I sync with iCal, which can then be set up to send me an email reminding me to return the book or renew. Very handy.

Never used Journler (sp?) so I can’t comment on that. I also have Yojimbo, but I use that for serial numbers, activation codes, etc. I use DTPro to store a large amount of research, and have it set up to pretty much mirror my folder hierarchy in Finder.

15

Yes I agree with all suggestions above. Journler’s drop box is really neat on the desktop - wish Scrivener had one.

You would have to take a serious look at MacJournal too.

I have used it for all my writing stuff for ages. It works really smoothly with Scrivener.

marinersoftware.com/

Of course it also works really well with Mariner Write and Montage.

:slight_smile:

16

Please give EagleFiler another try. Version 1.2 is much faster.

17

The problem with all these information managers is that you in a way have to marry them. Whatever synergetical powers they offer, they will unfold only if you put ALL your stuff into them. And ALL, that’s a lot.

On the PC, I have worked for several years with askSam, a freeform database system with an awkward interface, but with stupendous searching capablities - I still have to meet something alike on the Mac. Nevertheless, my overall rating of this adventure is rather negative. I do not count the costs (although askSam is very expensive), the trouble I had to put some things in (it had strange difficulties with common formats like HTML, did not provide essential features like tables and so on), but I count the effect on the long run: When I decided to switch to the Mac, it was an almost impossible task to get everything (or at least the most of it) OUT again. I worked half a week, had to write WORDBASIC-Makros to cut the immense export file into single items, and what remained are around 10.000 files named “Info 1.rtf”, “Info 2.rtf” and so on. And of course all interlinks I’ve created are lost.

Even if I stayed on the PC, trouble was ahead: I had bought version 5. Soon afterwards, version 6 was published. I was reluctant to upgrade - and did good to do so, because version 6 was reported as dreadfully buggy. (Even version 5 was buggy enough: I lost one database completely. “File not readable”. Okay - and then? I didn’t suceed to open any of my backuped files either and finally let the mystery unrevolved, because the file had not been that important anyway…) No matter which PIM you choose, there is no garantie that it will never go that way also. Software companies get sold, break down, change their strategy, portfolio, whatever. The company that makes DevonThink (which appears to me like the best alternative to what I wanted with askSam) appears rock-solid from what can be found in the Internet, but will they still be like this in five, in ten years?

In the moment, being freshly divorced from my former PIM (although it was no happy relationship), I am reluctant to bind myself to another. In the moment, I bet on open file formats. If you have mostly RTF, HTML and TXT, you can do whatever you want. And for someone who comes from the PC, Spotlight is still a dream of a feature.

There are even times when I think the only really reliable information collection I own are my handwritten notes. Paper survives centuries. I still have and can read notes I took 30, 40 years ago. Sometimes I think I should just print out and file everything thats important… :frowning:

There once was a guy named Niklas Luhman, a sociologist who is famous for having spend all his life creating a huge, huge card index - tens of thousands of cards, hand-written notes, excerpts from books he read, thoughts, whatever, and all linked to each other with a clever and endlessly extensible system. He wrote all his books by wandering through his cards, and he explained that his system had grown to such an extent that it was able to surprise him again and again, igniting new ideas and connections between subjects. Fascinating, isn’t it?

First thing one thinks is: If only he had used a computer! The possibilities! Automatic processing, linking, listing! Scanning for keywords! And so on.

But then I think: If he had used a computer, all he had put into it would most probably be lost today. It would be in a file format unreadable today, the company that provided the software would not exist anymore - whatever.

18

[quote=“AndreasE”]
The problem with all these information managers is that you in a way have to marry them. Whatever synergetical powers they offer, they will unfold only if you put ALL your stuff into them. And ALL, that’s a lot.

Isn’t the point with Eagle Filer that you don’t have to marry it? That all your information remains in its native format, and Eagle Filer simply knows where to look for it and how to display it?

I speak here not as an Eagle Filer user but as a current Mori user who’s looking to jump-ship, and who simply doesn’t have the time to weather the learning shoals on Devon-Think Pro. (I tried.)

19

Yes, maybe, but I still don’t understand what EagleFiler does what I cannot do with Spotlight and a doubleclick instead a single click on any file found. I have my nested folders in the file system etc. (BTW, I forgot to mention in my post above: Before using askSam, I simply collected downloaded files (HTML, RTF, DOC etc.) and put them in a huge nested system of subfolders for subjects and sub-sub-folders for sub-subjects: All these information is still alive and well and moved to my Mac within a couple of minutes!)

The minimum I’d want from a information system would be to show me the phrase I am looking for in the context (which askSam did, among other possibilities (useful was “show me the first paragraf of each document”)) so that I can more easily decide whether the corresponding document might be what I am looking for or not. And who knows, maybe Spotlight will do this in a further incarnation?

20

Whew! Lots of great stuff here (and indeed, quite a bit more than I expected). So, because working on being productive is MUCH more fun than actually BEING productive, here’s what I’ve come up with. I tried almost every program mentioned in this article, and I think I’ve narrowed it down. Keep in mind, all of what I say here is based upon my own workflow, so take anything I say with a grain of salt. My only hope is that someone out there works like I do.

First, in terms of usage, I find that tagging actually occurs later in my workflow. Tagging is something I do in Scrivener, once I’ve extracted what I need from my collection of information, so I can quickly access it whilst writing. Otherwise, I end up with an endlessly-propagating tagset and that’s of no help to anyone. This means that strong searching capabilities are a MUST. Also, it seems like a goal of these programs is to get away from the Finder metaphor with zillions of folders with further zillions of subdocuments.

The Awesome, but Not for Me
Yojimbo – Too rigid, databasey, not quite for me. No nested folders (this makes my inner taxonomy freak cringe).

EagleFiler – Not a big fan of its tagging system, slower than alternatives, and its storage mechanism annoys me (the metric TON of .plist files). Just didn’t hit me right on that visceral level.

=============

The Contenders
Journler – I LIKE this program. It’s slick-looking, stable. The nested smart-folders as a sort of standing & operator are ridiculously freaking cool. Tabbing, bookmarks bar. It makes it ludicrously easy to have plaintext notes and, e.g., PDF’s in the same “entry.” All that said, sometimes its “Journal” metaphor gets in the way. If I have a PDF in there, I don’t want to look at an empty pane, then click the PDF a second time as if it were an attachment. On the bright side, it doesn’t use databases! On the down side again, it’s MUCH slower than some of the competition, especially on the importing. But once stuff is imported, it flies as fast anything else. Oh, and it’s free, and it’s spotlight-aware (this is both a positive and a negative).

DevonThink Pro – You know, I think I’m going to end up using this program almost by default. Yeah, it’s databasey, and its interface is clunky (God, some of those viewing options are stupid). But it is FAST, and its searching capabilities are kind of awesome. Since I’m not a big tagging-guy, I don’t really miss the lack. And besides, I figured out a way around that – just create a folder called “!Tags” (so it appears at the top of the list), with subfolders for each “Tag.” Then stick replicants (not duplicates, more like aliases) of your files in each Tag. Not perfect, but not awful either. And its searching is AMAZING. I really am impressed. It also seems to churn out the best search results of any of the software out there. I’d love to wait for DTP 2.0, but who knows when that’s coming out. One major complaint, though. Drag in a plain text file, and it’s stored in the structure of the database itself. That makes it impossible to drag out to something like, say, Scrivener.

=============

Added bonus: my favorite Finder/Organization trick. The word Aardvark. Seriously. It’s the first word in the dictionary, so in any alphabetical listing, it will appear at the top, regardless of whatever else is in the folder. Also, it’s a fairly uncommon word to use as a filename, so it’s immediately recognizable. I use it as a place to drop immediate workin’ things, so I don’t have to dig through a folder to find them.