A Better Comment Script

A couple of years ago, I presented a shell script for commenting out lines, for use in a LaTeX mode for SubEthaEdit. The script is an improvement over the AppleScript approach used in other SubEthaEdit modes, but does something I don't really like: it always inserts or removes comments at the beginning of the lines, rather than at an appropriate indentation level.

Below, I present line-comment, an awk script that handles line-oriented comments, taking indentation into account. Lines to comment or uncomment are read from Standard Input, and the processed lines are written to Standard Output. The script uses the current commenting of the lines to determine whether to comment or uncomment.

There are two options that can be set. First, there is the TabWidth, which defines an indentation level; this defaults to the (basically useless) Unix standard of an eight-space tab. You'll almost always want to set this, even if you're using tabs, not spaces, for indentation. Second, there is the CommentString, whose meaning should be obvious; this defaults to the hash character # common to many programming languages.

As an example, a nice choice for Python could be line-comment TabWidth=4, while for Scala line-comment TabWidth=2 CommentString="//" would be more suitable.

Update: If the script doesn't seem to work, try setting the environment variable COMMAND_MODE=unix2003. This is relevant if you want to call it from SubEthaEdit: SEE uses COMMAND_MODE=legacy, which can cause the regular expressions to fail to match.

The script is unchanged:

#! /usr/bin/awk -f

function trimIndent(text, indRE,   n, tokens) {
  # Returns the text with the indentation removed. Sets
  # global variable IndentLevel to show how many
  # indentation levels were removed.
  n = split(text, tokens, indRE)
  if (n > 1) {
    IndentLevel = n-1
    rest = tokens[n]
  } else {
    IndentLevel = 0
    rest = text
  }
  return rest
}

function commentIndex(txt, commtxt,      n) {
  n = index(txt, commtxt)
  if (n > 0 && substr(txt, 1, n-1) !~ /^[ \t]*$/) {
    n = 0
  }
  return n
}

function noncommentPrefix(txt) {
  return match(txt, /^[ \t]*/) ? substr(txt, 1, RLENGTH) : ""
}

function multiString(str, mult,     n, mstr) {
  mstr = ""
  for (n=0; n<mult; n++) {
    mstr = mstr str
  }
  return mstr
}

function offsetString(offset) {
  return multiString(" ", offset)
}

BEGIN {
  TabWidth=8
  CommentString="#"
}

NR == 1 {
  # Establish regex based on tab settings. This comes after
  # the BEGIN to allow the TabWidth to be overriden.
  indentRegex = "( {0,"(TabWidth-1)"}\t| {"TabWidth"})"
  # indentRegex = "( {0,3}\t|"offsetString(TabWidth)")"
}

{
  # Common processing for all lines. Divide the line into a
  # prefix of whitespace, followed immediately by the
  # comment string, if present, or a non-tab, non-space
  # character if not. The prefix consists of indentation
  # steps followed by an offset, defined as a number of spaces
  # insufficient to constitute an indentation step.
  Line[NR] = $0
  commInd = commentIndex($0, CommentString)
  if (commInd > 0) {
    prefix = substr($0, 1, commInd-1)
    CommentPosition[NR] = commInd
  } else {
    prefix = noncommentPrefix($0)
  }
  offset = length(trimIndent(prefix, indentRegex))
}

NR == 1 {
  BaseOffset = offset
  BaseIndent = IndentLevel
  MinOffset = BaseOffset
  MinIndent = BaseIndent
}

NR > 1 {
  if (IndentLevel < MinIndent ) {
    MinIndent = IndentLevel
    MinOffset = 0
  } else if (offset < MinOffset) {
    MinOffset = offset
  }
}

END {
  commLen = length(CommentString)
  if (length(CommentPosition) == length(Line) && MinIndent == BaseIndent && MinOffset == BaseOffset) {
    for (n=1; n<=NR; n++) {
      commPos = CommentPosition[n]
      print substr(Line[n], 1, commPos-1) substr(Line[n], commLen+commPos)
    }
  } else {
    indentPart = MinIndent ? indentRegex"{"MinIndent"}" : ""
    # indentPart = multiString(indentRegex, MinIndent)
    offsetPart = offsetString(MinOffset)
    # offsetPart = offsetString(MinOffset)
    commRegex = "^" indentPart offsetPart
    for (n=1; n<=NR; n++) {
      match(Line[n], commRegex)
      print substr(Line[n], 1, RLENGTH) CommentString substr(Line[n], 1+RLENGTH)
    }
  }
}

SEEing LaTeX 29: Listening to LaTeX

The SubEthaEdit mode I've developed in this series of posts is generally rather quiet. It doesn't announce successful production of a PDF from a LaTeX file, it just creates it and opens it. It also doesn't tell you when an error occurs.

You can have the errors reported by piping the output from the call to pdflatex into some command line tool. Exactly what tool to use is less clear. The natural choice for SubEthaEdit would be the see command line too, but it's not very convenient, and has some real problems. Beyond that, calling out to see repeatedly will produce a new window each time, which is just annoying.

The Python mode has a nicer behavior. It has a Check Syntax command that always writes to a window named Python Check Syntax, opening it if need be or overwriting the contents otherwise. The Lua mode has an analogous Check Syntax command with similar behavior.

Let's make a shell script that does something similar. First, I encapsulate the relevant bits of AppleScript from the Python and Lua modes into some simple handlers. Second, I embed those into a shell script using the approach recently discussed here. The resulting script I call seeless, as it is usable as something like the less pager. I'll defer the text of the script until the end, first describing the usage.

Basic usage is just to pipe in some text:
echo Hello World! | seeless
This writes the text Hello World into a SubEthaEdit window titled seeless message, opening a new window if necessary or replacing the text in an existing window. Multiple windows with the same title are a bit problematic, with no guarantee that the window you want will be the one written to. Don't do that.

The title of the window can be specified:
echo Hello World! | seeless -t"A message for you, direct from the shell"
Appropriate choice of title allows a SubEthaEdit mode to establish its own reporting window.

The window is normally brought to the front when it is written to. Call seeless as
echo Hello World! | seeless -b
to leave the window in the background. This allows, e.g., a reporting window to be kept out of the way as a tab and only checked when something seems to be wrong.

There are two different modes, insert and append, for writing to the window. For insert mode, the window is cleared before writing the text from stdin, while append mode just appends to any existing text. To set append more, use:
echo Hello World! | seeless -a
Insert mode is the default, but a flag exists for it, too:
echo Hello World! | seeless -i
Multiple flags for the insert and append mode can be given, with the final one determining the behavior.

In append mode, the text from a second call to seeless follows immediately after the text from the first. To give some visual space, specify a separator:
echo Hello World! | seeless -s"----------"
When a separator is given with the -s flag, append mode is automatically set. It is also possible to use a separator in insert mode, giving a form of header. For example,
echo Hello World! | seeless -s"$(date)" -i
shows the time when seeless was called. Another possibility would be to have a cluster of related programs writing to the same window, and using the separator to specify the program that wrote the latest text.

Let's take a look at how I put those options into use with the LaTeX mode. I set SEE_BIBTEX to:

'bibtex "${FILE%.tex}" | "$HOME/Library/bin/seeless" -t"LaTeX Messages" -s"bibtex ran at $(date)\n" -b -i &> /dev/null &'

SEE_LATEX_CLEANUP to:

'latexmk -C "$FILE" | "$HOME/Library/bin/seeless" -t"LaTeX Messages" -s"latexmk -C ran at $(date)\n" -b -i &> /dev/null &'

and SEE_LATEX_COMPILER to:

'latexmk -pdf -quiet "$FILE" | "$HOME/Library/bin/seeless" -t"LaTeX Messages" -s"latexmk ran at $(date)\n" -b -i &> /dev/null &'

Note that I have redirected the output of each call to seeless to /dev/null and made the calls asynchronous with &—SubEthaEdit hangs without doing this, requiring a force quit.

With the above settings, the LaTeX mode will cause SubEthaEdit to open up a report window titled LaTeX Messages whenever a document is typeset, bibtex is run, or the auxiliary files are cleaned up. I can put it out of the way, either as a tab or background document; because I've used the -b flag for each call, the report window will stay out of the way until I want it brought to the front. I've used a separator to show which feature was most recently used and at what time it was called. I've set insert mode with an -i flag, so I only see the latest call; by eliminating this flag, I'd have a chronological record of all the calls made (in the current editing session, anyway).

I haven't been using this very long, so there may be some bugs. However, it seems quite solid, and is definitely useful already. Download it here.

As promised above, I'll give the text of the script here, too. I've formatted the script as a shell script, to better show how the shell variables are used to adapt the behavior of the embedded AppleScript.

#!/bin/sh

# Writes stdin to a SubEthaEdit document, modifying the contents if the
# document already exists. Document is selected by title, with a default
# title of 'seeless'. Title can be specified with a '-t' flag. If there
# are multiple documents with the given title, one is chosen arbitrarily.
#
# Writing to the document occurs in two modes, insert and append. With
# insert mode, the document is cleared before any text is written to
# it. In append mode, any existing text is maintained, with the new
# text appended at the end. By giving an '-a' flag, append mode is set.
# Giving an '-i' flag sets insert mode; insert mode is the default.
#
# A separator can be specified. The separator is written to the
# document before the text from stdin. Specifying a separator also sets
# append mode (but this can be turned off again if desired with an '-i'
# flag). The separator is specified with an '-s' flag; the argument
# following the flag is used as the separator.
#
# Giving an '-h' flag shows a usage summary. Any other flags are ignored.
#

#$Id: seeless.sh,v 1.9 2008/04/12 20:33:56 mjb Exp $

# Copyright (c) 2008, Michael J. Barber
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject
# to the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
# IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR
# ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
# CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.


PROGRAM=$(basename $0)

usage()
{
    echo "Usage: $PROGRAM [-ahi] [-t title] [-s separator]"
}

title="$PROGRAM message"
shouldClear=true
shouldSeparate=false
shouldBringToFront=true
separator="----------"
while getopts :t:ias:bh opt
do
    case $opt in
    t)      title="$OPTARG"
            ;;
    i)      shouldClear=true
            ;;
    a)      shouldClear=false
            ;;
    s)      separator="$OPTARG"
            shouldSeparate=true
            shouldClear=false
            ;;
    b)      shouldBringToFront=false
            ;;
    h)      usage
            exit 0
            ;;
    '?')    echo "$PROGRAM: invalid option -$OPTARG" >&2
            usage >&2
            exit 1
            ;;
    esac
done

shift $((OPTIND - 1))

/usr/bin/osascript > /dev/null <<ASCPT
    set newContents to "$(cat | sed -e 's/\\/\\\\/g' -e 's/\"/\\\"/g')"
    set seeDoc to (ensureSEEDocumentExists for "$title")
    if $shouldClear then
        replaceContents for the seeDoc by ""
    end if
    if $shouldSeparate then
        extendContents for the seeDoc by the "$separator"
    end if
    extendContents for the seeDoc by the newContents
    if $shouldBringToFront then
        tell application "SubEthaEdit" to show the seeDoc
    end if
    
    to ensureSEEDocumentExists for doctitle
        tell application "SubEthaEdit"
            if exists document named doctitle then
                document named doctitle
            else
                make new document with properties {name:doctitle}
            end if
        end tell
    end ensureSEEDocumentExists
    
    to replaceContents for seeDoc by newContents
        tell application "SubEthaEdit"
            set the contents of seeDoc to newContents
            clear change marks of seeDoc
            try
                set modified of seeDoc to false
            end try
        end tell
    end replaceContents
    
    to extendContents for seeDoc by moreContents
        tell application "SubEthaEdit"
            if "" is not equal to the contents of the last paragraph of seeDoc then
                set the contents of the last insertion point of the last paragraph of seeDoc to return
            end if
            set the contents of the last insertion point of the last paragraph of seeDoc to moreContents
            clear change marks of seeDoc
            try
                set modified of seeDoc to false
            end try
        end tell
    end extendContents
ASCPT

A Bit More on AppleScript and stdin

I can get even simpler than the earlier script to pass stdin to an AppleScript embedding within a shell script. Consider this:

#!/bin/sh

/usr/bin/osascript > /dev/null <<ASCPT
    set stdinText to "$(cat | sed -e 's/\\/\\\\/g' -e 's/\"/\\\"/g')"
    tell application "TextEdit"
        activate
        make new document with properties {text:stdinText}
    end tell
ASCPT

No need for the temporary file anymore, just use cat from within the AppleScript portion. The result of cat needs to be piped through sed in order to prevent problems with quoting, with a bunch of backslashes to get the special characters right. This may need further tweaking.

I ran into this when I tried using the earlier script for showing the results of the various scripts for the SubEthaEdit LaTeX mode. The earlier script failed when used for showing the results of bibtex. For some reason that I've not been able to work out, the AppleScript Standard Additions are no longer reached and the call to do shell script fails. The new approach works, so far at least.

SEEing LaTeX 28: Some Critical Notes on the 'see' Command Line Tool

I've used the see command line tool to relate a LaTeX previewer to SubEthaEdit using pdfsync. Unfortunately, I've become aware that the natural usage of see has a definite problem. Plugging in see and -g %line "%file" for the "PDFSync support" preferences in Skim doesn't work quite the way one would hope.

With that usage, which I'd consider to be the most natural, you wind up producing a new see process each time you command-click in Skim to switch over to SubEthaEdit. That process, due to shortcomings in the design of see, will hang around until you close the document in SubEthaEdit. Of course, if you're using SubEthaEdit and Skim together like that, you're in the middle of editing a LaTeX document, so you're not likely to be closing the document very promptly. Do that enough, and you could consume all the user processes allowed by Mac OS X. If you've not experience running out of user processes, let's just describe it as Not Fun.

I don't think it is likely to be a big problem, especially with the more friendly process limits in Mac OS X Leopard, but it is definitely a real problem. The problem gets compounded with each new usage of see for integrating external applications or reporting from a mode script. End result is that some caution is warranted when using see, and it should be omitted in favor of another approach, if necessary. Personally, I'm still using it.

I've requested an enhancement to see consisting of a 'quiet' mode, like seen in, e.g., grep. That would eliminate the issue, and, as a side benefit, considerably simplify using see from AppleScripts for SubEthaEdit modes. Let's hope the Coding Monkeys act on it.

SEEing LaTeX 27: End in Sight

The activity here today, after a few weeks of silence, reflects that I've finished the documentation. I've spent some time earlier today getting everything ready. Finally, I am very pleased to announce that you can download the mode.

What comes next? That depends on you! Download it, try it out, and let me know about any problems, either by posting a comment or by sending me some email.

Update: The scripts developed in this series have been incorporated into the LaTeX mode distributed with SubEthaEdit. It should not normally be necessary to download the mode from the link given here, but I'll leave the link active in case anyone sees a need for my version.

Update 2: With SubEthaEdit 3.5, the LaTeX mode linked here is out of date, not supporting folding. You should not use it without good reason, as the mode that comes with SEE has everying this version does and more. I will leave the mode available for now, but I don't see much use for it.

SEEing LaTeX 26: Sample Environment Settings

As a starting point for customizing the LaTeX mode, here are some sample environment settings:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>BIBINPUTS</key>
    <string>"$HOME/Library/texmf/bibtex/bib"</string>
    <key>SEE_BIBTEX</key>
    <string>'bibtex "${FILE%.tex}" | open -f -a SubEthaEdit'</string>
    <key>SEE_LATEX_CLEANUP</key>
    <string>'latexmk -C "$FILE"'</string>
    <key>SEE_LATEX_COMMENT</key>
    <string>'"$SEE_MODE_RESOURCES"/bin/comment.sh "% "'</string>
    <key>SEE_LATEX_COMPILER</key>
    <string>'latexmk -pdf -quiet "$FILE"'</string>
    <key>SEE_LATEX_COMPILEVIEWER</key>
    <string>'open -a Skim "$PRODUCT"'</string>
    <key>SEE_LATEX_PRODUCT_TYPE</key>
    <string>pdf</string>
    <key>SEE_LATEX_VIEWER</key>
    <string>'export __CF_USER_TEXT_ENCODING=0x1F5:0:0; /Applications/Skim.app/Contents/SharedSupport/displayline $LINE "$PRODUCT"'</string>
</dict>
</plist>

SEEing LaTeX 25: Putting Things in Order

As noted below, the entries in the LaTeX mode menu for SubEthaEdit need to be put into a sensible order. After some thinking, I decided that there are really three groups of scripts: scripts for interacting with the LaTeX system, scripts for simplifying typing, and a script for interacting with the LaTeX mode itself.

Ideally, we'd put the three into groups divided by a horizontal rule. Unfortunately, SEE does not (yet) allow dividing lines to be inserted into the menu. Regardless, let's organize the scripts into the three groups, and - with one exception- just alphabetize within the groups. The exception is for the "Typeset and View" menu item; it strikes me as natural to put that first in the list.

End result is a menu ordered as:

Typeset and View
Clean Up Auxiliary Files
Run BibTeX
View

Complete Citation
Inline Math
Insert Environment...
Un/Comment Selected Lines

Customize Mode...

Although I've broken the groups apart with spacing, they'll just run together in the menu. Perhaps a later version of SEE will allow some more structure to be added.

One final change is that I've renamed the "Mode Environment..." menu item to "Customize Mode...". The two distinct meanings of "environment" struck me as confusing, and "Insert Environment..." is simply too appropriate for LaTeX to change.

I'll add another entry to the menu as soon as possible. It will go into the third group, with a title something like "Mode Help". I still have to write the mode help, first.

SEEing LaTeX 24: For Completeness, BibTeX

I guess the right solution is to add a menu item for running bibtex, and that's it. Since I don't use makeindex myself, I'll leave that aside, unless someone wants to contribute scripts or just examples of use. I'd guess that the scripts would be easy enough, just adapt the ones I'll present for BibTeX.

Here's the AppleScript:

checkSaveStatus without updating
set bibScript to join of {modeEnvironment(), quotedForm for "$SEE_MODE_RESOURCES/bin/runbibtex.sh", quotedForm for documentPath()} by space
do shell script bibScript

on seescriptsettings()
    return {displayName:"Run BibTeX"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

And here's the shell script that it calls:

#!/bin/sh

#$Id: runbibtex.sh,v 1.1 2008/01/06 19:09:49 mjb Exp $

PATH="$PATH:/usr/texbin:/usr/local/bin"
export PATH

BIBTEX=${SEE_BIBTEX:-'bibtex "$(basename $FILE .tex)"'}
FILE="$(basename "$1")"
DIRNAME="$(dirname "$1")"

cd "$DIRNAME"
eval $BIBTEX

SEEing LaTeX 23: Am I Done?

Last July, I listed some desirable features for the LaTeX mode for SubEthaEdit. The mentioned features were integrating with a PDF viewer using pdfsync, enabling insertion of citation keys in bibtex format, allow typesetting by calling pdflatex from within SEE, cleaning up auxiliary files, and commenting out selected lines. I also mentioned that it might be nice, if inessential, to be able to insert environments and formatting.

I've accomplished all of that. As well, I have introduced a mechanism for customizing the shell script environment for the mode and have developed a set of AppleScript handlers useful for scripting SEE. I'm quite pleased with how all of that has worked out. Not only do I now have a LaTeX mode that covers my main needs, but I've got a solid foundation on which I - and hopefully others! - can build support scripts for other modes.

That said, the LaTeX mode is not quite finished. It's clear that I should write some documentation, and add a "Mode Help" menu item. I should also better document SubEthaEditTools. Beyond that, there are two additional tasks.

First, the scripts in the LaTeX mode menu should be put into some sort of reasonable order, rather than the haphazard order that currently is there. The items appear in order based on the names of the scripts, which can differ from the entry in the menu. What is the right order to use?

Second, I've omitted some important elements of a LaTeX system, because latexmk handles them for me. For example, I have not provided a way to run bibtex. Should additional elements be added? If so, which? bibtex? makeindex? Something else?

SEEing LaTeX 22: Inline Math

Environments aren't the only thing common LaTeX constructs that are awkward to type. The delimiters for inline math are pretty awkward, too. Let's add those, too:

set mathText to selectionText()
set wrappedText to "\\( " & mathText & " \\)"
setSelectionText to wrappedText
if (length of mathText) equals 0 then
    set {startChar, nextChar} to selectionRange without extendingFront or extendingEnd
    setSelectionRange to (startChar + length of mathText + 3)
end if

on seescriptsettings()
    {displayName:"Inline Math", keyboardShortcut:"@~^m", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

SEEing LaTeX 21: Inserting Environments

Environments again? Yes, but this time we're going to look at environments in LaTeX, not the shell environment. Environments are a bit of a pain to type, but the repetitive structure makes them suitable to automation: we just get the name of the environment, and then have a more-or-less standard form:

\begin{environmentname}
    body
\end{environmentname}

Indentation of the body can be a little unclear. For example, I usually indent equation environments, but would never dream of indenting the document environment.

To add environment insertion into the LaTeX mode for SubEthaEdit, we begin by getting the environment name using display dialog:

try
    display dialog "Enter environment name:" with title "Insert Environment" default answer "equation"
on error number -128 -- user canceled
    return
end try
set envName to text returned of result

The try block is to handle when the user cancels instead of entering and environment name. Somewhat arbitrarily, I set the default answer to be "equation", since my guess is that equations are probably the most common environment.

After that, there is just some fiddly work getting the formatting of the environment correct. It needs to intelligently insert newlines and tabs to keep the document readable. As well, something needs to be done with the selection text. There are two possibilities as I see it: (1) treat the selection as the name of the environment and (2) treat the selection as the body of the environment. I went with the latter. Finally, it would be nice to place the insertion point somewhere reasonable; I think it works nicely at the end of the body, especially since the insertion point is positioned to start typing immediately if the body is empty. Putting it all together, we have:

set {startSelected, nextSelected} to selectionRange without extendingFront or extendingEnd
set {startExtended, nextExtended} to selectionRange with extendingFront and extendingEnd

set prefix to selectByComparing(startSelected, startExtended, "", "\n")
set suffix to selectByComparing(nextSelected, nextExtended, "", "\n")
set indent to selectByComparing(startSelected, nextSelected, "\t", "")

set beforeInsertion to (join of {prefix, "\\begin{", envName, "}\n", indent, selectionText()} by "")
set afterInsertion to (join of {suffix, "\\end{", envName, "}\n" } by "")

setSelectionText to (beforeInsertion & afterInsertion)
setSelectionRange to startSelected + (count of beforeInsertion) - 1 + (count of suffix)

Note that I've made repeated use of a convenience function:

to selectByComparing(val1, val2, sameVal, diffVal)
    if val1 equals val2 then
        sameVal
    else
        diffVal
    end if
end selectByComparing

The selectByComparing handler is not part of SubEthaEditTools - maybe it should be?

That's all there is to it, apart from the boilerplate:

on seescriptsettings()
    {displayName:"Insert Environment...", keyboardShortcut:"@^e", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

Update: There is an interesting possibility for adjusting the default environment. I had used "equation" as the default, but it was pretty arbitrary. Another approach would be to just repeat whichever environment was last given. We just add a property to the script that holds the default environment, use its value in making the dialog, and update the property based on the dialog result:

property defaultEnvironment: "equation"

try
    display dialog "Enter environment name:" with title "Insert Environment" default answer defaultEnvironment
on error number -128 -- user canceled
    return
end try
set envName to text returned of result
set defaultEnvironment to envName

Is this actually a good idea? or will it just be annoying? Hard to say without using it, so I guess I'll try it out for a while.

SEEing LaTeX 20: Getting Citations Right

I started enhancing the LaTeX mode for SubEthaEdit by looking at citations. The original approach, based on just using the input manager supplied with BibDesk, proved to be unsatisfactory. Let's fix that now.

The general approach seems clear enough. We need to determine a search term based on the cursor position in the LaTeX document, pass that to BibDesk to get matching documents, let the user pick which documents are relevant, format the selected documents, and insert the result into the document. Using SubEthaEditTools our SEE scripting abstraction layer, it proves to be fairly straightforward.

The first step, determining the search term, is probably the trickiest. What I envision is that the user can enter a partial citation and have it finished by the script. Thus, we'll need to examine the text immediately before the insertion point and determine a partial citation key. We should not cross an open brace "{", as we don't want to include the macro. Additionally, we shouldn't cross a comma, since we might be looking at multiple citations within one macro. Let's not check the calling macro; this allows the completion to be invoked at inappropriate points, but also allows the completion to be invoked for user-defined macros.

One final issue is what we do when there is a range of text selected. Since the default behavior should be to have no text selected, we should treat a non-empty selection as meaningful. Let's take it to mean that the search should be constrained to only include the selected text. Therefore, we determine a search term based either on the selected text or all the text on the line preceding the insertion point.

Putting all that together, I came up with:

set {startChar, nextChar} to selectionRange without extendingFront or extendingEnd
if startChar equals nextChar then
    -- empty selection, try the whole line
    set selectionContents to extendedSelectionText with extendingFront without extendingEnd
else
    set selectionContents to selectionText()
end if
set macroArgument to the last item of the (tokens of the selectionContents between "{")
set searchTerm to the last item of the (tokens of the macroArgument between ",")

I've used a couple of the new handlers from SubEthaEditTools. These are hopefully self-explanatory, but check the implementation in case they are not.

Using the two tokens calls, we obtain a partial citation key. We pass that to BibDesk:

tell application "BibDesk"
    set citeMatches to search for searchTerm with for completion
end tell

The ungrammatical with for completion will give us a list of cite keys, not just document titles. Each completion is given as a string containing both the cite key and some document information, separated by a " % " string.

We next present the list of completions to the user, in order to narrow the list down to just the appropriate citations. To present a list, we use choose from list from the AppleScript StandardAdditions. There are three cases worth considering. First, if there is only one publication, we can select it by default, so the user just confirms whether it is correct. Second, if there are multiple possible publications, we just show the list, declining to guess. Finally, if there aren't any matches, we just inform the user with display alert; in this case, we'll set the list of publications to the empty list. If there aren't any publications returned, that means the user canceled, so we should just exit and leave the document unchanged. Putting it all together, we arrive at:

if (count of citeMatches) equals 1 then
    choose from list citeMatches with title "Citation Matches" with prompt "One matching publication:" default items citeMatches
    set pubs to result
else if (count of citeMatches) > 1
    choose from list citeMatches with title "Citation Matches" with prompt "Please select publications:" with multiple selections allowed
    set pubs to result
else
    display alert "No matches found for partial citation \"" & searchTerm & "\""
    set pubs to {}
end if

if (count of pubs) equals 0 then
    -- user canceled, do nothing
    return
end

We now have a non-empty list of completions. We need to split those apart to get the cite keys, then join the cite keys with commas. I used awk:

set citation to shellTransform of (join of pubs by "\n") for "" through "awk -F' % ' 'NR == 1 { printf(\"%s\", $1) } NR > 1 { printf(\",%s\", $1) }'" without alteringLineEndings

Pretty ugly! Let's reformat the core of the awk program to make it clearer:

NR == 1 { printf("%s", $1) }
NR > 1 { printf(",%s", $1) }

In this form, it's clear enough, keeping in mind that we use " % " as the field separator: we just print out the first field, corresponding to the cite key, with the first record treated specially to get the number of commas right.

Finally, we insert the formatted citation back into the document. I also move the insertion point to the end of the formatted citation, as a typing convenience. I had considered closing the brace for the citation macro, but that would make multi-citation lists more awkward, so decided against it. This is none too difficult:

setSelectionRange to {nextChar - (length of searchTerm), nextChar - 1}
setSelectionText to citation
setSelectionRange to (nextChar - (length of searchTerm) + (length of citation))

Again, I've used new handlers from the SubEthaEditTools library.

Putting it all together, and adding a seescriptsettings handler to integrate it into SEE, we get:

-- $Id: BibDeskCompletions.applescript,v 1.2 2008/01/04 18:40:16 mjb Exp mjb $

(*
Need to figure out the search term. Treat a selection as meaning to constrain the search
term to lie within the selection, and an empty selection as meaning to get the search term
from the preceding text on the line. We don't cross an opening brace, so that the search term
comes from a call to a macro. However, we don't check to see if the macro is one of the
standard citation macros, since we do want to allow user macros.
*)
set {startChar, nextChar} to selectionRange without extendingFront or extendingEnd
if startChar equals nextChar then
    -- empty selection, try the whole line
    set selectionContents to extendedSelectionText with extendingFront without extendingEnd
else
    set selectionContents to selectionText()
end if
set macroArgument to the last item of the (tokens of the selectionContents between "{")
set searchTerm to the last item of the (tokens of the macroArgument between ",")

tell application "BibDesk"
    set citeMatches to search for searchTerm with for completion
end tell

-- get list of publications, customizing user interaction based on number of matches
if (count of citeMatches) equals 1 then
    choose from list citeMatches with title "Citation Matches" with prompt "One matching publication:" default items citeMatches
    set pubs to result
else if (count of citeMatches) > 1
    choose from list citeMatches with title "Citation Matches" with prompt "Please select publications:" with multiple selections allowed
    set pubs to result
else
    display alert "No matches found for partial citation \"" & searchTerm & "\""
    set pubs to {}
end if

if (count of pubs) equals 0 then
    -- user canceled, do nothing
    return
end

(*
At this point, there is a non-empty list of matches, which replaces the search term. By
construction, the search term always immediately precedes the end of the selection.
Call out to the shell to format the publication list into a LaTeX citation, insert the citation,
and then move the insertion point just after the citation.
*)
set citation to shellTransform of (join of pubs by "\n") for "" through "awk -F' % ' 'NR == 1 { printf(\"%s\", $1) } NR > 1 { printf(\",%s\", $1) }'" without alteringLineEndings
setSelectionRange to {nextChar - (length of searchTerm), nextChar - 1}
setSelectionText to citation
setSelectionRange to (nextChar - (length of searchTerm) + (length of citation))

on seescriptsettings()
    {displayName:"Complete Citation", shortDisplayName:"Citation", keyboardShortcut:"@^j",  toolbarIcon:"ToolbarBibDesk.png", inDefaultToolbar:"yes", toolbarTooltip:"Complete citation using BibDesk", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

SEEing LaTeX 19: SubEthaEditTools, or, Eating My Own Dogfood

Let's take the arguments in the last three posts seriously. We move all of the handlers developed in the course of the SEEing LaTeX series into a separate file called SubEthaEditTools.applescript, and rewrite all of the AppleScripts to include the file using m4. While we're at it, we also add some more handlers to the SubEthaEditTools to try to make a more complete set, with the goal being to write scripts for the mode by using the handlers without directly interacting with SubEthaEdit in the AppleScript.

This requires relatively minor changes to the makefile I've been using, but nothing too serious. Also, I construct SubEthaEditTools itself using m4, with more general purpose handlers for, e.g., string manipulation placed in their own files. With the more modular structure, some additional minor rewrites of some handlers seems appropriate, which do not warrant specific comment.

A concept that is made explicit in the SubEthaEditTools is that of the extended selection. That is, the selection modified either so that the beginning is extended to the start of the first line of the selection, or to the end of the last line or the selection, or both. The extended selection has appeared implicitly several times, so it seems worthwhile to make it explicit. Further, there are now handlers both for extending the selection (forward or backward) and for referring to the extended selection without modifying the actual selection.

Update: I've replaced the quotedForm handler with a doubleQuotedForm handler, to help prevent confusion with the quoted form of action for AppleScript strings. I've also worked in a few usages of quoted form of in the various scripts for the LaTeX mode (I didn't know about quoted form of until recently).

Also, I've added a license. It's an MIT-style license, so should be suitably permissive for use by others.

Update 2 (2008/04/23): I've added a few more handlers. Specifically, modeSetting, selectionIsEmpty, and documentIsAvailable.

-- SubEthaEdit Tools

(*
Copyright (c) 2008, Michael J. Barber

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject
to the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR
ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

*)


-- Environment management

on modeEnvironment()
    join of {"export __CF_USER_TEXT_ENCODING=0x1F5:0x8000100:0x8000100;", "export SEE_MODE_RESOURCES=", doubleQuotedForm for modeResources(), "; ", readEnvironment out of environmentFilePath()} by ""
end modeEnvironment

on modeSetting for envVar
    try
        tell application "System Events"
            tell property list file (my environmentFilePath())
                get value of the property list item envVar
            end tell
        end tell
    on error
        missing value
    end try
end modeEnvironmentSetting

to openEnvironmentSettings()
    openEnvironment at environmentFilePath() with settingDefaultEnvironment
end openEnvironment

on environmentFilePath()
    tell application "SubEthaEdit" to set modeName to name of the mode of the front document
     join of {path to preferences from user domain as string, "de.codingmonkeys.SubEthaEdit.", modeName, "_environment.plist"} by ""
end environmentFileName

-- Manipulation of document text

on documentText()
    tell the front document of application "SubEthaEdit" to get the contents
end documentText

on selectionIsEmpty()
    tell the front document of application "SubEthaEdit" to get the length of the selection
    result is equal to 0
end selectionIsEmpty

to completeSelectedLines()
    extendSelection with extendingFront and extendingEnd
end completeSelectedLines

on selectionText()
    tell the front document of application "SubEthaEdit" to get the contents of the selection
end selectionText

to setSelectionText to newText
    tell application "SubEthaEdit" to set the contents of the selection of the front document to the newText
end setSelectionText

on selectionRange given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    tell the front document of application "SubEthaEdit"
        if shouldExtendFront and shouldExtendEnd then
            get {startCharacterIndex of the first paragraph of the selection, nextCharacterIndex of the last paragraph of the selection}
        else if shouldExtendFront then
            get {startCharacterIndex of the first paragraph of the selection, nextCharacterIndex of the selection}
        else if shouldExtendEnd then
            get {startCharacterIndex of the selection, nextCharacterIndex of the last paragraph of the selection}
        else
            get {startCharacterIndex of the selection, nextCharacterIndex of the selection}
        end if
    end tell
end selectionRange

to setSelectionRange to newRange
    tell the front document of application "SubEthaEdit"
        set selection to newRange
    end tell
end setSelectionRange

on extendedSelectionText given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    set {startChar, nextChar} to selectionRange given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    tell the front document of application "SubEthaEdit"
        get the contents of characters startChar through (nextChar - 1) as text
    end tell
end extendedSelectionText

to extendSelection given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    set {startChar, nextChar} to (selectionRange given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd)
    setSelectionRange to {startChar, nextChar-1}
end extendSelection

-- Manipulation of document properties

to checkSaveStatus given updating:shouldSave
    tell application "SubEthaEdit"
        if not (exists path of front document) then
            error "You have to save the document first"
        end if
        if shouldSave and (modified of front document) then
            try
                save front document
            end try
        end if
    end tell
end checkSaveStatus

on documentIsAvailable()
    tell application "SubEthaEdit" to get the count of the documents
    result > 0
end documentIsAvailable

to requireNewlineAtEOF()
    tell the front document of application "SubEthaEdit"
        if "" is equal to the contents of the last paragraph then
            -- final line terminated, do nothing
        else
            set the contents of the last insertion point of the last paragraph to return
        end if
    end tell
end requireNewlineAtEOF

on documentPath()
    tell application "SubEthaEdit" to get the path of the front document
end documentPath

on documentLine()
    tell application "SubEthaEdit" to get the startLineNumber of selection of front document
end documentLine

on modeResources()
    tell application "SubEthaEdit" to get the resource path of the mode of the front document
end modeResources

-- String Utilities

on replacement of oldDelim by newDelim for sourceString
    return join of (tokens of sourceString between oldDelim) by newDelim
end replacement

on tokens of str between delimiters
    set oldTIDs to text item delimiters of AppleScript
    set text item delimiters of AppleScript to delimiters
    set strtoks to text items of str
    set text item delimiters of AppleScript to oldTIDs
    return strtoks
end tokens

on join of tokenList by delimiter
    set oldTIDs to text item delimiters of AppleScript
    set text item delimiters of AppleScript to delimiter
    set joinedString to tokenList as string
    set text item delimiters of AppleScript to oldTIDs
    return joinedString
end join

on doubleQuotedForm for baseString  
    quote & baseString & quote
end doubleQuotedForm

on shellTransform of inText for envString through pipeline given alteringLineEndings:altEnds
    set shellscript to join of {envString, "pbpaste", "|", pipeline} by space
    set the oldClipboard to the clipboard
    set the clipboard to the inText
    try
        set shellresponse to do shell script shellscript altering line endings altEnds
    on error errMsg number errNum from badObject
        set the clipboard to the oldClipboard
        error errMsg number errNum from badObject
    end try
    set the clipboard to the oldClipboard
    shellresponse
end shellTransform

-- Handling of environment settings using a plist file

to writeDefaultEnvironment at envPath
    set savedClipboard to the clipboard
    set the clipboard to "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<!DOCTYPE plist PUBLIC \"-//Apple Computer//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">
<plist version=\"1.0\">
<dict/>
</plist>"
    try
        do shell script "pbpaste > " & (POSIX path of envPath)
    on error errMsg number errNum from badObject
        set the clipboard to the savedClipboard
        error errMsg number errNum from badObject
    end try
    set the clipboard to the savedClipboard
end writeDefaultEnvironment

to openEnvironment at envFilePath given settingDefaultEnvironment:shouldSetDefault
    tell application "System Events"
        if not exists file envFilePath
            if shouldSetDefault
                my writeDefaultEnvironment at envFilePath
            else
                error ("Can't get environment file " & quote & envFilePath & quote) number -1728
            end if
        end if
        open file envFilePath
    end tell
end openEnvironment

to readEnvironment out of plist
    readListPair out of plist
    environmentString from result
end readEnvironment

to readListPair out of plist
    tell application "System Events"
        if exists file plist then
            tell property list file plist
                get {name, value} of every property list item
            end tell
        else
            {{}, {}}
        end if
    end tell
end readPlist

on environmentString from keyValueListPair
    set {plistKeys, plistValues} to keyValueListPair
    set accumulator to {}
    set oldTIDs to text item delimiters of AppleScript
    set text item delimiters of AppleScript to ""
    repeat with i from 1 to number of items in plistKeys
        set tokens to {"export ", item i of plistKeys, "=", item i of plistValues, ";"}
        copy (tokens as string) to the end of the accumulator
    end repeat
    set AppleScript's text item delimiters to space
    set envString to accumulator as string
    set AppleScript's text item delimiters to oldTIDs
    envString
end environmentString