SEEing LaTeX 20: Getting Citations Right

I started enhancing the LaTeX mode for SubEthaEdit by looking at citations. The original approach, based on just using the input manager supplied with BibDesk, proved to be unsatisfactory. Let's fix that now.

The general approach seems clear enough. We need to determine a search term based on the cursor position in the LaTeX document, pass that to BibDesk to get matching documents, let the user pick which documents are relevant, format the selected documents, and insert the result into the document. Using SubEthaEditTools our SEE scripting abstraction layer, it proves to be fairly straightforward.

The first step, determining the search term, is probably the trickiest. What I envision is that the user can enter a partial citation and have it finished by the script. Thus, we'll need to examine the text immediately before the insertion point and determine a partial citation key. We should not cross an open brace "{", as we don't want to include the macro. Additionally, we shouldn't cross a comma, since we might be looking at multiple citations within one macro. Let's not check the calling macro; this allows the completion to be invoked at inappropriate points, but also allows the completion to be invoked for user-defined macros.

One final issue is what we do when there is a range of text selected. Since the default behavior should be to have no text selected, we should treat a non-empty selection as meaningful. Let's take it to mean that the search should be constrained to only include the selected text. Therefore, we determine a search term based either on the selected text or all the text on the line preceding the insertion point.

Putting all that together, I came up with:

set {startChar, nextChar} to selectionRange without extendingFront or extendingEnd
if startChar equals nextChar then
    -- empty selection, try the whole line
    set selectionContents to extendedSelectionText with extendingFront without extendingEnd
else
    set selectionContents to selectionText()
end if
set macroArgument to the last item of the (tokens of the selectionContents between "{")
set searchTerm to the last item of the (tokens of the macroArgument between ",")

I've used a couple of the new handlers from SubEthaEditTools. These are hopefully self-explanatory, but check the implementation in case they are not.

Using the two tokens calls, we obtain a partial citation key. We pass that to BibDesk:

tell application "BibDesk"
    set citeMatches to search for searchTerm with for completion
end tell

The ungrammatical with for completion will give us a list of cite keys, not just document titles. Each completion is given as a string containing both the cite key and some document information, separated by a " % " string.

We next present the list of completions to the user, in order to narrow the list down to just the appropriate citations. To present a list, we use choose from list from the AppleScript StandardAdditions. There are three cases worth considering. First, if there is only one publication, we can select it by default, so the user just confirms whether it is correct. Second, if there are multiple possible publications, we just show the list, declining to guess. Finally, if there aren't any matches, we just inform the user with display alert; in this case, we'll set the list of publications to the empty list. If there aren't any publications returned, that means the user canceled, so we should just exit and leave the document unchanged. Putting it all together, we arrive at:

if (count of citeMatches) equals 1 then
    choose from list citeMatches with title "Citation Matches" with prompt "One matching publication:" default items citeMatches
    set pubs to result
else if (count of citeMatches) > 1
    choose from list citeMatches with title "Citation Matches" with prompt "Please select publications:" with multiple selections allowed
    set pubs to result
else
    display alert "No matches found for partial citation \"" & searchTerm & "\""
    set pubs to {}
end if

if (count of pubs) equals 0 then
    -- user canceled, do nothing
    return
end

We now have a non-empty list of completions. We need to split those apart to get the cite keys, then join the cite keys with commas. I used awk:

set citation to shellTransform of (join of pubs by "\n") for "" through "awk -F' % ' 'NR == 1 { printf(\"%s\", $1) } NR > 1 { printf(\",%s\", $1) }'" without alteringLineEndings

Pretty ugly! Let's reformat the core of the awk program to make it clearer:

NR == 1 { printf("%s", $1) }
NR > 1 { printf(",%s", $1) }

In this form, it's clear enough, keeping in mind that we use " % " as the field separator: we just print out the first field, corresponding to the cite key, with the first record treated specially to get the number of commas right.

Finally, we insert the formatted citation back into the document. I also move the insertion point to the end of the formatted citation, as a typing convenience. I had considered closing the brace for the citation macro, but that would make multi-citation lists more awkward, so decided against it. This is none too difficult:

setSelectionRange to {nextChar - (length of searchTerm), nextChar - 1}
setSelectionText to citation
setSelectionRange to (nextChar - (length of searchTerm) + (length of citation))

Again, I've used new handlers from the SubEthaEditTools library.

Putting it all together, and adding a seescriptsettings handler to integrate it into SEE, we get:

-- $Id: BibDeskCompletions.applescript,v 1.2 2008/01/04 18:40:16 mjb Exp mjb $

(*
Need to figure out the search term. Treat a selection as meaning to constrain the search
term to lie within the selection, and an empty selection as meaning to get the search term
from the preceding text on the line. We don't cross an opening brace, so that the search term
comes from a call to a macro. However, we don't check to see if the macro is one of the
standard citation macros, since we do want to allow user macros.
*)
set {startChar, nextChar} to selectionRange without extendingFront or extendingEnd
if startChar equals nextChar then
    -- empty selection, try the whole line
    set selectionContents to extendedSelectionText with extendingFront without extendingEnd
else
    set selectionContents to selectionText()
end if
set macroArgument to the last item of the (tokens of the selectionContents between "{")
set searchTerm to the last item of the (tokens of the macroArgument between ",")

tell application "BibDesk"
    set citeMatches to search for searchTerm with for completion
end tell

-- get list of publications, customizing user interaction based on number of matches
if (count of citeMatches) equals 1 then
    choose from list citeMatches with title "Citation Matches" with prompt "One matching publication:" default items citeMatches
    set pubs to result
else if (count of citeMatches) > 1
    choose from list citeMatches with title "Citation Matches" with prompt "Please select publications:" with multiple selections allowed
    set pubs to result
else
    display alert "No matches found for partial citation \"" & searchTerm & "\""
    set pubs to {}
end if

if (count of pubs) equals 0 then
    -- user canceled, do nothing
    return
end

(*
At this point, there is a non-empty list of matches, which replaces the search term. By
construction, the search term always immediately precedes the end of the selection.
Call out to the shell to format the publication list into a LaTeX citation, insert the citation,
and then move the insertion point just after the citation.
*)
set citation to shellTransform of (join of pubs by "\n") for "" through "awk -F' % ' 'NR == 1 { printf(\"%s\", $1) } NR > 1 { printf(\",%s\", $1) }'" without alteringLineEndings
setSelectionRange to {nextChar - (length of searchTerm), nextChar - 1}
setSelectionText to citation
setSelectionRange to (nextChar - (length of searchTerm) + (length of citation))

on seescriptsettings()
    {displayName:"Complete Citation", shortDisplayName:"Citation", keyboardShortcut:"@^j",  toolbarIcon:"ToolbarBibDesk.png", inDefaultToolbar:"yes", toolbarTooltip:"Complete citation using BibDesk", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

SEEing LaTeX 4: More on Completions

Earlier, I took a look at using BibDesk to get SubEthaEdit to complete citations in LaTeX files. After working with it a few days, I've learned a few more things about BibDesk. Unfortunately, not all good.

There seems to be a rather unpleasant interaction between BibDesk's completions and SubEthaEdit's completions. Normally, SubEthaEdit injects some mode-dependent terms into the completion dictionary. For some modes, like Python, this is no big deal; most of the keywords in Python are normal English words, so they would be completed by the standard completion dictionary. For other modes, e.g., the Objective-C mode, it can make a more significant contribution; the Objective-C mode defines a large number of Cocoa class names, which you can see by typing NS and pressing F5. Also, and maybe more importantly, SubEthaEdit will search through the current file to find matches, so, e.g., variables you've defined will be completed. For LaTeX, that provides a way to have the labels for equations, figures, and the like to be completed; for user-defined macros to be easily referenced; and for names and technical terms that don't appear in the standard dictionaries to be completed.

Using BibDesk's Autocompletion input manager disables SubEthaEdit's Autocompletion features. Apparently, any input manager that extends completions will have this effect, as the SubEthaEdit FAQ notes that TextExtras will stop SubEthaEdit's Autocompletions from working. With TextExtras, that wouldn't be so bad, since it provides completions of its own, but the tradeoff with BibDesk is more extreme. You can have citations and, interestingly, the \ref and \pageref commands for cross-references completed, but lose completion of the terms in the document, whether it be LaTeX or something else. Or, you can have autocompletion of the terms in the document, but lose autocompletion of citations (cross-reference completion is handled just fine by completion of terms from the document).

An early idea on how to resolve this was that I could install TextExtras, and use its completions. I used to use it, and liked it a lot, but gave up on it some time ago. I'm not sure exactly why, now, but I think it was because an earlier version of TextExtras conflicted with the then-new autocompletions in Mac OS X 10.3 Panther. Unfortunately, I haven't been able to connect to the TextExtras page to download it since coming up with the idea!

In the end, I disabled BibDesk's Autocompletion input manager, it simply comes at too high of a cost. BibDesk has a number of system services, which allow citations to be inserted that way. It will have to do for now. A solution using AppleScript to get completions from BibDesk looks possible, but it seems best to make the issue clear now and solve it later.

This is definitely a strike against using SubEthaEdit for LaTeX.

Update: The same thing happens when BibDesk's input manager is loaded into other editors, too. In Smultron, you lose completion of words from the document; I'm not so familiar with Smultron, so I don't know if there are other features that get lost. It is truly a shame that one of the coolest features of BibDesk is also one that I just don't want to use.

SEEing LaTeX 1: Citations

As a first step towards turning SubEthaEdit into a capable LaTeX editing machine, let's address citations. Citations are a challenge to keep consistent and accurate, so mechanized assistance is quite useful. For LaTeX, that means BibTeX. BibTeX separates content from presentation in much the way that TeX does. Considering the many different formatting styles for bibliographies and citations that are required by different journals, this is something that I won't give up on -- I'll type the citation keys by hand if needed.

Fortunately, getting citation support in SubEthaEdit is trivial! Just install BibDesk, if you're not already using it, and set up autocompletion for SubEthaEdit. Inside a \cite, hit F5 or command-escape, and you get a list of completions for a partial citation key. This works really well. I hadn't used BibDesk for that for a while, since TextMate had its own method (not that it has worked in recent versions), so I'd forgotten how slick this is.

Completions in general, not just for citations, can be made a little bit easier. In your user Library folder, there may be a KeyBindings folder; if not, create it. Create a file DefaultKeyBinding.dict in the KeyBindings folder. The file is an XML file with a list of key-action pairs; bind the escape key to a string "complete:" (with the colon, without the quotes). The Property List Editor works nicely for doing this. The end result is that you just press escape instead of command-escape to get a completion. It doesn't cause any problems for using escape to, say, select cancel in dialogs, so there's no down side, just completions that are easier to activate.