SEEing LaTeX 20: Getting Citations Right

I started enhancing the LaTeX mode for SubEthaEdit by looking at citations. The original approach, based on just using the input manager supplied with BibDesk, proved to be unsatisfactory. Let's fix that now.

The general approach seems clear enough. We need to determine a search term based on the cursor position in the LaTeX document, pass that to BibDesk to get matching documents, let the user pick which documents are relevant, format the selected documents, and insert the result into the document. Using SubEthaEditTools our SEE scripting abstraction layer, it proves to be fairly straightforward.

The first step, determining the search term, is probably the trickiest. What I envision is that the user can enter a partial citation and have it finished by the script. Thus, we'll need to examine the text immediately before the insertion point and determine a partial citation key. We should not cross an open brace "{", as we don't want to include the macro. Additionally, we shouldn't cross a comma, since we might be looking at multiple citations within one macro. Let's not check the calling macro; this allows the completion to be invoked at inappropriate points, but also allows the completion to be invoked for user-defined macros.

One final issue is what we do when there is a range of text selected. Since the default behavior should be to have no text selected, we should treat a non-empty selection as meaningful. Let's take it to mean that the search should be constrained to only include the selected text. Therefore, we determine a search term based either on the selected text or all the text on the line preceding the insertion point.

Putting all that together, I came up with:

set {startChar, nextChar} to selectionRange without extendingFront or extendingEnd
if startChar equals nextChar then
    -- empty selection, try the whole line
    set selectionContents to extendedSelectionText with extendingFront without extendingEnd
else
    set selectionContents to selectionText()
end if
set macroArgument to the last item of the (tokens of the selectionContents between "{")
set searchTerm to the last item of the (tokens of the macroArgument between ",")

I've used a couple of the new handlers from SubEthaEditTools. These are hopefully self-explanatory, but check the implementation in case they are not.

Using the two tokens calls, we obtain a partial citation key. We pass that to BibDesk:

tell application "BibDesk"
    set citeMatches to search for searchTerm with for completion
end tell

The ungrammatical with for completion will give us a list of cite keys, not just document titles. Each completion is given as a string containing both the cite key and some document information, separated by a " % " string.

We next present the list of completions to the user, in order to narrow the list down to just the appropriate citations. To present a list, we use choose from list from the AppleScript StandardAdditions. There are three cases worth considering. First, if there is only one publication, we can select it by default, so the user just confirms whether it is correct. Second, if there are multiple possible publications, we just show the list, declining to guess. Finally, if there aren't any matches, we just inform the user with display alert; in this case, we'll set the list of publications to the empty list. If there aren't any publications returned, that means the user canceled, so we should just exit and leave the document unchanged. Putting it all together, we arrive at:

if (count of citeMatches) equals 1 then
    choose from list citeMatches with title "Citation Matches" with prompt "One matching publication:" default items citeMatches
    set pubs to result
else if (count of citeMatches) > 1
    choose from list citeMatches with title "Citation Matches" with prompt "Please select publications:" with multiple selections allowed
    set pubs to result
else
    display alert "No matches found for partial citation \"" & searchTerm & "\""
    set pubs to {}
end if

if (count of pubs) equals 0 then
    -- user canceled, do nothing
    return
end

We now have a non-empty list of completions. We need to split those apart to get the cite keys, then join the cite keys with commas. I used awk:

set citation to shellTransform of (join of pubs by "\n") for "" through "awk -F' % ' 'NR == 1 { printf(\"%s\", $1) } NR > 1 { printf(\",%s\", $1) }'" without alteringLineEndings

Pretty ugly! Let's reformat the core of the awk program to make it clearer:

NR == 1 { printf("%s", $1) }
NR > 1 { printf(",%s", $1) }

In this form, it's clear enough, keeping in mind that we use " % " as the field separator: we just print out the first field, corresponding to the cite key, with the first record treated specially to get the number of commas right.

Finally, we insert the formatted citation back into the document. I also move the insertion point to the end of the formatted citation, as a typing convenience. I had considered closing the brace for the citation macro, but that would make multi-citation lists more awkward, so decided against it. This is none too difficult:

setSelectionRange to {nextChar - (length of searchTerm), nextChar - 1}
setSelectionText to citation
setSelectionRange to (nextChar - (length of searchTerm) + (length of citation))

Again, I've used new handlers from the SubEthaEditTools library.

Putting it all together, and adding a seescriptsettings handler to integrate it into SEE, we get:

-- $Id: BibDeskCompletions.applescript,v 1.2 2008/01/04 18:40:16 mjb Exp mjb $

(*
Need to figure out the search term. Treat a selection as meaning to constrain the search
term to lie within the selection, and an empty selection as meaning to get the search term
from the preceding text on the line. We don't cross an opening brace, so that the search term
comes from a call to a macro. However, we don't check to see if the macro is one of the
standard citation macros, since we do want to allow user macros.
*)
set {startChar, nextChar} to selectionRange without extendingFront or extendingEnd
if startChar equals nextChar then
    -- empty selection, try the whole line
    set selectionContents to extendedSelectionText with extendingFront without extendingEnd
else
    set selectionContents to selectionText()
end if
set macroArgument to the last item of the (tokens of the selectionContents between "{")
set searchTerm to the last item of the (tokens of the macroArgument between ",")

tell application "BibDesk"
    set citeMatches to search for searchTerm with for completion
end tell

-- get list of publications, customizing user interaction based on number of matches
if (count of citeMatches) equals 1 then
    choose from list citeMatches with title "Citation Matches" with prompt "One matching publication:" default items citeMatches
    set pubs to result
else if (count of citeMatches) > 1
    choose from list citeMatches with title "Citation Matches" with prompt "Please select publications:" with multiple selections allowed
    set pubs to result
else
    display alert "No matches found for partial citation \"" & searchTerm & "\""
    set pubs to {}
end if

if (count of pubs) equals 0 then
    -- user canceled, do nothing
    return
end

(*
At this point, there is a non-empty list of matches, which replaces the search term. By
construction, the search term always immediately precedes the end of the selection.
Call out to the shell to format the publication list into a LaTeX citation, insert the citation,
and then move the insertion point just after the citation.
*)
set citation to shellTransform of (join of pubs by "\n") for "" through "awk -F' % ' 'NR == 1 { printf(\"%s\", $1) } NR > 1 { printf(\",%s\", $1) }'" without alteringLineEndings
setSelectionRange to {nextChar - (length of searchTerm), nextChar - 1}
setSelectionText to citation
setSelectionRange to (nextChar - (length of searchTerm) + (length of citation))

on seescriptsettings()
    {displayName:"Complete Citation", shortDisplayName:"Citation", keyboardShortcut:"@^j",  toolbarIcon:"ToolbarBibDesk.png", inDefaultToolbar:"yes", toolbarTooltip:"Complete citation using BibDesk", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

SEEing LaTeX 19: SubEthaEditTools, or, Eating My Own Dogfood

Let's take the arguments in the last three posts seriously. We move all of the handlers developed in the course of the SEEing LaTeX series into a separate file called SubEthaEditTools.applescript, and rewrite all of the AppleScripts to include the file using m4. While we're at it, we also add some more handlers to the SubEthaEditTools to try to make a more complete set, with the goal being to write scripts for the mode by using the handlers without directly interacting with SubEthaEdit in the AppleScript.

This requires relatively minor changes to the makefile I've been using, but nothing too serious. Also, I construct SubEthaEditTools itself using m4, with more general purpose handlers for, e.g., string manipulation placed in their own files. With the more modular structure, some additional minor rewrites of some handlers seems appropriate, which do not warrant specific comment.

A concept that is made explicit in the SubEthaEditTools is that of the extended selection. That is, the selection modified either so that the beginning is extended to the start of the first line of the selection, or to the end of the last line or the selection, or both. The extended selection has appeared implicitly several times, so it seems worthwhile to make it explicit. Further, there are now handlers both for extending the selection (forward or backward) and for referring to the extended selection without modifying the actual selection.

Update: I've replaced the quotedForm handler with a doubleQuotedForm handler, to help prevent confusion with the quoted form of action for AppleScript strings. I've also worked in a few usages of quoted form of in the various scripts for the LaTeX mode (I didn't know about quoted form of until recently).

Also, I've added a license. It's an MIT-style license, so should be suitably permissive for use by others.

Update 2 (2008/04/23): I've added a few more handlers. Specifically, modeSetting, selectionIsEmpty, and documentIsAvailable.

-- SubEthaEdit Tools

(*
Copyright (c) 2008, Michael J. Barber

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject
to the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR
ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

*)


-- Environment management

on modeEnvironment()
    join of {"export __CF_USER_TEXT_ENCODING=0x1F5:0x8000100:0x8000100;", "export SEE_MODE_RESOURCES=", doubleQuotedForm for modeResources(), "; ", readEnvironment out of environmentFilePath()} by ""
end modeEnvironment

on modeSetting for envVar
    try
        tell application "System Events"
            tell property list file (my environmentFilePath())
                get value of the property list item envVar
            end tell
        end tell
    on error
        missing value
    end try
end modeEnvironmentSetting

to openEnvironmentSettings()
    openEnvironment at environmentFilePath() with settingDefaultEnvironment
end openEnvironment

on environmentFilePath()
    tell application "SubEthaEdit" to set modeName to name of the mode of the front document
     join of {path to preferences from user domain as string, "de.codingmonkeys.SubEthaEdit.", modeName, "_environment.plist"} by ""
end environmentFileName

-- Manipulation of document text

on documentText()
    tell the front document of application "SubEthaEdit" to get the contents
end documentText

on selectionIsEmpty()
    tell the front document of application "SubEthaEdit" to get the length of the selection
    result is equal to 0
end selectionIsEmpty

to completeSelectedLines()
    extendSelection with extendingFront and extendingEnd
end completeSelectedLines

on selectionText()
    tell the front document of application "SubEthaEdit" to get the contents of the selection
end selectionText

to setSelectionText to newText
    tell application "SubEthaEdit" to set the contents of the selection of the front document to the newText
end setSelectionText

on selectionRange given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    tell the front document of application "SubEthaEdit"
        if shouldExtendFront and shouldExtendEnd then
            get {startCharacterIndex of the first paragraph of the selection, nextCharacterIndex of the last paragraph of the selection}
        else if shouldExtendFront then
            get {startCharacterIndex of the first paragraph of the selection, nextCharacterIndex of the selection}
        else if shouldExtendEnd then
            get {startCharacterIndex of the selection, nextCharacterIndex of the last paragraph of the selection}
        else
            get {startCharacterIndex of the selection, nextCharacterIndex of the selection}
        end if
    end tell
end selectionRange

to setSelectionRange to newRange
    tell the front document of application "SubEthaEdit"
        set selection to newRange
    end tell
end setSelectionRange

on extendedSelectionText given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    set {startChar, nextChar} to selectionRange given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    tell the front document of application "SubEthaEdit"
        get the contents of characters startChar through (nextChar - 1) as text
    end tell
end extendedSelectionText

to extendSelection given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    set {startChar, nextChar} to (selectionRange given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd)
    setSelectionRange to {startChar, nextChar-1}
end extendSelection

-- Manipulation of document properties

to checkSaveStatus given updating:shouldSave
    tell application "SubEthaEdit"
        if not (exists path of front document) then
            error "You have to save the document first"
        end if
        if shouldSave and (modified of front document) then
            try
                save front document
            end try
        end if
    end tell
end checkSaveStatus

on documentIsAvailable()
    tell application "SubEthaEdit" to get the count of the documents
    result > 0
end documentIsAvailable

to requireNewlineAtEOF()
    tell the front document of application "SubEthaEdit"
        if "" is equal to the contents of the last paragraph then
            -- final line terminated, do nothing
        else
            set the contents of the last insertion point of the last paragraph to return
        end if
    end tell
end requireNewlineAtEOF

on documentPath()
    tell application "SubEthaEdit" to get the path of the front document
end documentPath

on documentLine()
    tell application "SubEthaEdit" to get the startLineNumber of selection of front document
end documentLine

on modeResources()
    tell application "SubEthaEdit" to get the resource path of the mode of the front document
end modeResources

-- String Utilities

on replacement of oldDelim by newDelim for sourceString
    return join of (tokens of sourceString between oldDelim) by newDelim
end replacement

on tokens of str between delimiters
    set oldTIDs to text item delimiters of AppleScript
    set text item delimiters of AppleScript to delimiters
    set strtoks to text items of str
    set text item delimiters of AppleScript to oldTIDs
    return strtoks
end tokens

on join of tokenList by delimiter
    set oldTIDs to text item delimiters of AppleScript
    set text item delimiters of AppleScript to delimiter
    set joinedString to tokenList as string
    set text item delimiters of AppleScript to oldTIDs
    return joinedString
end join

on doubleQuotedForm for baseString  
    quote & baseString & quote
end doubleQuotedForm

on shellTransform of inText for envString through pipeline given alteringLineEndings:altEnds
    set shellscript to join of {envString, "pbpaste", "|", pipeline} by space
    set the oldClipboard to the clipboard
    set the clipboard to the inText
    try
        set shellresponse to do shell script shellscript altering line endings altEnds
    on error errMsg number errNum from badObject
        set the clipboard to the oldClipboard
        error errMsg number errNum from badObject
    end try
    set the clipboard to the oldClipboard
    shellresponse
end shellTransform

-- Handling of environment settings using a plist file

to writeDefaultEnvironment at envPath
    set savedClipboard to the clipboard
    set the clipboard to "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<!DOCTYPE plist PUBLIC \"-//Apple Computer//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">
<plist version=\"1.0\">
<dict/>
</plist>"
    try
        do shell script "pbpaste > " & (POSIX path of envPath)
    on error errMsg number errNum from badObject
        set the clipboard to the savedClipboard
        error errMsg number errNum from badObject
    end try
    set the clipboard to the savedClipboard
end writeDefaultEnvironment

to openEnvironment at envFilePath given settingDefaultEnvironment:shouldSetDefault
    tell application "System Events"
        if not exists file envFilePath
            if shouldSetDefault
                my writeDefaultEnvironment at envFilePath
            else
                error ("Can't get environment file " & quote & envFilePath & quote) number -1728
            end if
        end if
        open file envFilePath
    end tell
end openEnvironment

to readEnvironment out of plist
    readListPair out of plist
    environmentString from result
end readEnvironment

to readListPair out of plist
    tell application "System Events"
        if exists file plist then
            tell property list file plist
                get {name, value} of every property list item
            end tell
        else
            {{}, {}}
        end if
    end tell
end readPlist

on environmentString from keyValueListPair
    set {plistKeys, plistValues} to keyValueListPair
    set accumulator to {}
    set oldTIDs to text item delimiters of AppleScript
    set text item delimiters of AppleScript to ""
    repeat with i from 1 to number of items in plistKeys
        set tokens to {"export ", item i of plistKeys, "=", item i of plistValues, ";"}
        copy (tokens as string) to the end of the accumulator
    end repeat
    set AppleScript's text item delimiters to space
    set envString to accumulator as string
    set AppleScript's text item delimiters to oldTIDs
    envString
end environmentString

What Role Should the Script Editor Play in Writing AppleScript?

I think it is safe to say that Script Editor is the most widely used application for writing AppleScripts. But, should it be? I'm coming to the conclusion that it brings so many limitations along with it that it can only play a limited role.

Before going any further with this line of thought, I'd like to make clear that I'm not making any recommendation for or against the various commercial AppleScript development environments. Looking at the SEEing LaTeX series, you might get the impression that I use AppleScript a fair amount. I don't. I use it as little as possible, which is why in that series I do most of the real work not in AppleScript, but in shell scripts called from AppleScript. Paying for a commercial environment may make sense if you want to or have to write a lot of AppleScript, but would be a waste of money for me.

However, our choices are not limited to Script Editor and the commercial development environments. We have an excellent development environment that comes for free with Mac OS X: it's called Unix. We can use make to manage a project, of which AppleScript can play a useful part. Unfortunately, Script Editor is tied too strongly to AppleScript, which means we're more or less stuck with AppleScript's mechanisms for modularity. Since modularity goes hand in hand with reuse, this is a key point.

Within AppleScript, the usual approach is to put common functionality in a script library and use load script from the StandardAdditions. This has a number of problems. First, there isn't a standard search path from which to load the script library, so you need to introduce a new convention. Second, if a library is changed, there is no automated mechanism to update scripts that depend on it. You can load the script library, store it in a property, and just check to see if the library has been updated whenever the script is called; this roughly emulates how, e.g., import works in Python. There are enough steps to this process that you're looking at a lot of copy and paste programming. Also, you depend on whatever convention you've established to load the script library.

Consider now using make. You can use osacompile to compile your text source into a compiled script. You could also use various shell methods to manipulate your script to define the search path for the script libraries, but how is the Script Editor to work with that? You can also just forget about reloading the script libraries when they change, since make will take care of that for us. We just load the script library once, and forget about it. Since we're just loading the script library once, why not just forget about the search path while we're at it? Just compile the library source into your working directory, and have the script load it from that same directory.

At this point, we still can use Script Editor to edit the script sources. However, we still have a lot of boilerplate to write for little gain. We need to set up the scripts correctly so that make can run them once to load the library. We need to add extra steps to make in order to load the libraries. We need to refer to the script libraries by using the property in which we stored them, whenever we want to use a handler in the library.

Alternatively, instead of emulating how Python loads modules, let's take a look at how something like C would do things. In C, you just use the preprocessor to include the various files. We can do that for AppleScript using, e.g., m4. Assuming you're working in a script called demo.applescript, and want to use some handlers you've written for manipulating text. Just include the relevant library source with something like

include(`StringTools.applescript')

and compile with

m4 demo.applescript | osacompile -o demo.scpt

Just like in C, this approach includes everything into the namespace of the script. And, just like in C, this could lead to namespace conflicts. So far, I've only used this in a few scripts, all of which are relatively small. I've not encountered any difficulties. But, should trouble arise, there is a solution. Instead of including the text at the top level of the script, include it inside of a script object:

script StringTools
    include(`StringTools.applescript')
end script

This could be further automated by writing an m4 macro, such as:

define(`import', `script `$2'
include(`$1')
end script
')dnl

I'm not at all sure that this approach would scale to larger scripts well. You'd really like to be able to have scripts import what they need, and not worry about whether they will themselves be imported into others. Since I'm not planning to write any large AppleScripts, I'm not going to worry about that until it becomes a problem.

Where does this leave the Script Editor? It's no longer useful for scripts of any length, since it can't handle the include statements. We do want it in order to look at scripting dictionaries, that much is clear, but is it needed for any editing at all? It is a reasonable environment in which to try out little code snippets, but we can even dispose of that, since Script Editor offers that functionality through the services menu.

SEEing LaTeX 18: Cutting SubEthaEdit out of the Picture

Let's take further steps along the path begun last time. Recall that we moved the portion of the AppleScript that dealt with the SubEthaEdit mode into the handler that produces the environment for the shell scripts. Let's introduce a few more handlers:

-- Manipulation of document properties

to checkSaveStatus given updating:shouldSave
    tell application "SubEthaEdit"
        if not (exists path of front document) then
            error "You have to save the document first"
        end if
        if shouldSave and (modified of front document) then
            try
                save front document
            end try
        end if
    end tell
end checkSaveStatus

on documentPath()
    tell application "SubEthaEdit" to get the path of the front document
end documentPath

on documentLine()
    tell application "SubEthaEdit" to get the startLineNumber of selection of front document
end documentLine

The checkSaveStatus handler is the most complex of the three, and the only one that warrants any discussion. It never returns a meaningful value, and thus is only to be called for its side effects. There are two possible side effects. First, if the front document in SEE has never been saved, an error is raised. Second, if requested and necessary, the handler will update the file on disk by saving the current document.

That may seem a bit obscure, so let's examine an example. The AppleScript for typesetting previously featured a rather complicated nesting of tell, try, and if blocks. Using the new handlers, the only thing outside the handlers in a rewritten typesetting script is:

checkSaveStatus with updating
set buildScript to join of {modeEnvironment(), quotedForm for "$SEE_MODE_RESOURCES/bin/buildlatex.sh", quotedForm for documentPath(), documentLine()} by space
do shell script buildScript

on seescriptsettings()
    return {displayName:"Typeset and View", shortDisplayName:"Typeset", keyboardShortcut:"@b", toolbarIcon:"ToolbarIconBuildAndRun", inDefaultToolbar:"yes", toolbarTooltip:"Typeset and view the current document", inContextMenu:"no"}
end seescriptsettings

Just three statements, plus the seescriptsettings handler to connect it to SEE.

The various handlers introduced thus far have been to support several actions in the LaTeX mode: typesetting, viewing the product PDF in an external viewer, cleaning up auxiliary files, and commenting out lines. All of these actions can be rewritten using just the handlers, without directly addressing SubEthaEdit at all.

The handlers thus provide a useful abstraction layer for working with several distinct types of actions. They definitely won't cover every case of interest, but should still be useful patterns for a lot of common actions in many different modes.

SEEing LaTeX 17: A Bit More on Environments

To make the comments scripts a little more flexible, it turns out that we need to make some adjustments to how we handle the shell environment. The changes needed are minor, but, as we'll see, they lead to some new possibilities.

First, let's take a look at why the change is needed. I'd like to have more flexibility in how I call the comment.sh shell script. Significantly, I'd like to be able to call something else first, perhaps tr to change line endings to the newlines ("\n") that the shell requires. Alternatively, the shell script could be replaced with another tools for applying comments, much like how the typesetting scripts call pdflatex by default, but can be replaced by another tool like latexmk.

Doing that is quite straightforward. We just write a little shell script to handle the defaults, just like we did with other actions for the mode. It's just two lines:

#!/bin/sh -

COMMENT=${SEE_LATEX_COMMENT:-'"$SEE_MODE_RESOURCES/bin/comment.sh" %'}

eval $COMMENT

Notice that the default call needs to know the path to the resources directory for the mode, indicated here as the shell variable SEE_MODE_RESOURCES. Also, I've abandoned the original meaning of the SEE_LATEX_COMMENT variable; now, it defines the complete behavior for the comment command, not just the string to use.

Perhaps the simplest way to make the path available is just to define SEE_MODE_RESOURCES with the appropriate value. To do that, we rewrite the modeEnvironment handler to inject the appropriate value. I came up with this:

on modeEnvironment()
    tell application "SubEthaEdit" to set {modeName, modeResources} to {name, resource path} of the mode of the front document
    set envFilePath to (path to preferences from user domain as string) & "de.codingmonkeys.SubEthaEdit." & modeName & "_environment.plist"
    join of {"export SEE_MODE_RESOURCES=", quotedForm for modeResources, "; ", readEnvironment out of envFilePath} by ""
end modeEnvironment

I've also eliminated the parameter to the handler, which had specified the language mode for SubEthaEdit. The mode details were only used in defining the environment, so it seemed sensible.

With the new modeEnvironment handler, the logic for the CommentLines AppleScript becomes quite simple:

set env to modeEnvironment()
set pipeline to quotedForm for "$SEE_MODE_RESOURCES/bin/commentlines.sh"

completeSelectedLines()
set outText to shellTransform of selectionText() for env through pipeline without alteringLineEndings
setSelectionText to outText

There is now no direct interaction with SEE; the program is written, in effect, in a domain specific language for scripting SEE, abstracting away from the details of SEE's scripting implementation.

As a final point, I'd like to emphasize that the resource path for the mode is now available to the user. For example, I prefer to have a space after the percent sign for LaTeX comments. Thus, I just define SEE_LATEX_COMMENT to be '"$SEE_MODE_RESOURCES"/bin/comment.sh "% "' in the mode environment.

SEEing LaTeX 16: Comments Continued

In installment 15 of this series, I discussed difficulties that I'd encountered while exploring adding comments to the SubEthaEdit LaTeX mode. In so doing, I presented a shell script and several AppleScript handlers to address the difficulties. With that infrastructure, it is not so hard to incorporate comments into the mode.

First, there is an additional, minor change. Earlier, I'd described a prependEnvironment AppleScript handler. This proves to be too specific. A slightly simpler modeEnvironment handler seems to be a better fit. It is defined as:

on modeEnvironment for seeMode
    set envFilePath to (path to preferences from user domain as string) & "de.codingmonkeys.SubEthaEdit." & (name of seeMode) & "_environment.plist"
    readEnvironment out of envFilePath
end modeEnvironment

Hopefully, that is straightforward enough.

With the infrastructure set up, the logic for the AppleScript becomes pretty straightforward. We first get some needed information from the SEE document. We use that information to read out the environment using modeEnvironment and to build a pipeline string calling our comment.sh shell script. Then, we adjust the text selection to select complete lines, transform that text by running it through pipeline, and set the selection to the transformed text. Additionally, we define seescriptsettings to integrate our AppleScript into SEE. The settings are patterned after those of the Objective C mode. Here is the relevant fragment of the AppleScript:

tell application "SubEthaEdit"
        set activeMode to mode of front document
        set modeResources to resource path of activeMode
end tell

set env to modeEnvironment for activeMode

completeSelectedLines()
set inText to selectionText()
set pipeline to join of {quotedForm for (modeResources & "/bin/comment.sh"), quotedForm for "${SEE_LATEX_COMMENT:-%}"} by space  

set outText to shellTransform of inText for env through pipeline without alteringLineEndings
setSelectionText to outText

-- SubEthaEdit settings

on seescriptsettings()
    {displayName:"Un/Comment Selected Lines", keyboardShortcut:"@/", inContextMenu:"yes"}
end seescriptsettings

Note that the comment string can be set with the SEE_LATEX_COMMENT environment variable, defaulting to a "%".

The result is fairly nice, but not without shortcomings. There is a brief, but noticable, hesitation between activating the script and seeing the results. It's reasonably clear that building up the environment string is the problem; I'm not sure whether it's worth fixing or not. More significantly, there is a subtle logic problem. Implicitly, I've assumed that the LaTeX document is using Unix-style ("\n") line endings. That's a pretty safe assumption, most of the time, but can be wrong; we'll fix that next time.

SEEing LaTeX 15: Comments on Comments

I really thought that getting comments working in the SubEthaEdit LaTeX mode would be easy. It wasn't.

My expectation was that I could just adapt the Un/Comment Selected Lines script from the Objective C mode. However, I didn't really find the script to be satisfactory. Aesthetically, it's unappealing. Instead of using all the selected lines to determine whether to comment or uncomment the lines, just the first line is used. You can see the comments being applied one line at a time, because the AppleScript implementation loops over the lines, with each iteration in the loop sending its own, slow AppleEvent to SEE.

More importantly, the script functions in a manner that I consider to simply be incorrect. If lines are already commented, the objc mode script leaves them unaltered. Now consider working with a block of lines, only some of which are commented. If you invoke the Un/Comment Selected Lines script twice, the first invocation will comment, and the second will uncomment. Since the first invocation leaves the originally commented lines unchanged, there is no distinction between them and the originally uncommented lines. The second invocation thus removes the comments from all the lines, failing to restore the original state, changing the meaning of the program. In Objective C, that probably leads to an error at compile time. In LaTeX, you've just altered your document in a way that is quite likely to still be valid. It's also quite likely to be wrong, wrong, wrong!

In short, I decided I needed to rewrite the Un/Comment Selected Lines script to be (1) more aesthetically pleasing and (2) its own inverse. Working directly in AppleScript was not so easy. To eliminate the loop that causes the aesthetic issues, you really need to use a where clause that addresses all the lines at once. I couldn't get that to work. Maybe someone who's better with AppleScript could. But, really, why put in the effort, when the shell is considerably more powerful for text manipulation?

Once again, I'd figured it would be easy. Just a little grep to detect whether I should comment or uncomment and a little sed to make the actual change. After actually trying it, I quickly realized that there were some real challenges in trying to dynamically build up the regular expressions needed for grep and sed. As I often do when scripting, I found awk to be the solution to my problem, producing comment.sh:

#!/bin/sh

# Toggle comment status for text lines. Text lines are read from
# stdin and un/commented text lines are written to stdout.
#
# Comments are defined by the line starting with a text string given
# by the first argument. The lines will be uncommented if all lines
# are commented, and commented if any or all of the the lines are
# uncommented. The script is its own inverse, i.e., piping the text
# through the script twice writes the original text to stdout.

#$Id: comment.sh,v 1.3 2007/12/18 21:40:47 mjb Exp $

tmp=$(mktemp /tmp/comments.XXXXXXXXXXXXXXXXXXXX)

clen=$(printf "%s" "$1" | wc -c)

tee "$tmp" |
    awk -v clen="$clen" '{ print substr($0, 1, clen) }' |
        grep -F -q -v "$1"

if (($?))
then
    # uncomment
    cat "$tmp" | awk -v lnbeg=$((clen+1)) '{ print substr($0, lnbeg) }'
else
    # comment
    cat "$tmp" | awk -v comment="$1" '{ print comment $0 }'
fi

trap "rm -f $tmp; exit" EXIT HUP INT TERM

The script reads lines from stdin, writing either commented or uncommented lines to stdout. The comment string is given by the first argument to the script.

In comment.sh, I first make a temporary file, since I'll need to go through the lines twice; using tee, I can make a copy of the lines in the temp file. I determine the length of the comment string. I then use awk to cut away just the first few characters of each line, comparing them to the comment string with grep -F (-F for fixed strings, no regular expressions!). That determines whether or not all lines start with the comment string.

When all the lines start with the comment string, I uncomment by chopping off the initial characters using awk. Otherwise, I comment by printing both the comment string and the lines, again using awk. Again, note that I've avoided using regular expressions.

The last line ensures that the temp file is removed. There's not much else to say about it.

With the comment.sh script available, it now remains to get the necessary text from the LaTeX document and send it to the script. I follow the basic strategy shown on the Coding Monkeys website, consisting of copying the text to the clipboard and using pbpaste to pipe it into a desired shell script. I wrapped all that up into an AppleScript handler:

on shellTransform of inText for envString through pipeline given alteringLineEndings:altEnds
    set shellscript to envString & " export __CF_USER_TEXT_ENCODING=0x1F5:0x8000100:0x8000100; pbpaste | " & pipeline
    set the oldClipboard to the clipboard
    set the clipboard to the inText
    try
        set shellresponse to do shell script shellscript altering line endings altEnds
    on error errMsg number errNum from badObject
        set the clipboard to the oldClipboard
        error errMsg number errNum from badObject
    end try
    set the clipboard to the oldClipboard
    shellresponse
end shellTransform

Note the use of the try block to restore the clipboard in case of error, followed by another statement restoring the clipboard. It should then be that the clipboard is always restored to its original state. This construction is a little awkward, so the handler is a natural abstraction to hide the mess. The environment variable __CF_USER_TEXT_ENCODING follows the Coding Monkeys site example exactly - it doesn't seem to hurt when I omit it, but I'll just trust that it is correct.

What remains is to specify exactly what the text is. At first glance, it seems like we should just take the selected text. This has a serious drawback: you need to completely select all the lines you're interested in, or you'll add comments to the middle of a line. As an important special case, you'd be unable to just press the keyboard shortcut to comment out the current line with no selected text. So, I decided to extend the selection to complete the first and last lines of the selection; the special case is handled cleanly in this way, too. I defined another handler to manage the selection:

to completeSelectedLines()
    tell the front document of application "SubEthaEdit"
        set {startChar, nextChar} to {startCharacterIndex of paragraph (startLineNumber of selection), nextCharacterIndex of paragraph (endLineNumber of selection)}
        set selection to {startChar, nextChar - 1}
    end tell
end completeSelectedLines

The handler is a little complicated to understand because of how it is written. An equivalent form is:

to completeSelectedLines2()
    tell the front document of application "SubEthaEdit"
        set startLineNum to startLineNumber of the selection
        set endLineNum to endLineNumber of the selection
        set startChar to startCharacterIndex of paragraph startLineNum
        set nextChar to nextCharacterIndex of paragraph endLineNum
        set selection to {startChar, nextChar - 1}
    end tell
end completeSelectedLines2

The second form is a lot slower, though, because it sends individual AppleEvents to handle things that are done in just one with the first set statement of the first form.

With that, all the infrastructure needed to put together the desired Un/Comment Selected Lines script are at hand. I'll do that in the next installment.

Update: A better shell script for handling the comments is available.