Ctags in SubEthaEdit

We've now looked at how to locate the right tags file and match a tag against it by working in the shell. But our goal is to connect Ctags to an editor, SubEthaEdit (SEE) in this case. We thus will need to switch from the world of the shell to the world of AppleScript. In this post, I'll just focus on getting the path to the tags file and a tag for which to search from SEE.

I'll not be working directly with SubEthaEdit's AppleScript dictionary, instead using my SubEthaEditTools handlers as a basis. Should anyone be interested in connecting Ctags to another Mac OS X editor that supports AppleScript, it would probably be better to port the SubEthaEditTools handlers to work with the editor and directly use the scripts I'll present here.

As a general design strategy, I'll identify two AppleScript error numbers with expected behaviors. First, I'll use number 901 to indicate that tag processing should be abandoned. Second, I'll use number 902 to indicate that an error of known type has occurred. This lets me handle a broad class of troubles by either quietly exiting, or beeping then exiting. Any other errors will just be unhandled, causing SubEthaEdit to show a sheet with details of the error.

Additionally, I'll need to define a search path for shell tools. Rather than using a customizable environment as I've done before, I'll just define one as an AppleScript property:

property UnixPath : "export PATH=\"$HOME/Library/Application Support/SubEthaEdit/bin:/Library/Application Support/SubEthaEdit/bin:$HOME/Library/bin:/usr/local/bin:/opt/local/bin:/usr/bin:/bin:/usr/local/sbin:/opt/local/sbin:/usr/sbin:/sbin\";"

To find the tags file, I first need to make sure a document is available to use as the starting point for the search. Second, I just need to call out to the shell with an appropriate command. Encapsulating these in handlers, I define:

on requireValidDocumentForCtags()
  if not documentIsAvailable() then
    error "No document open" number 902
  end if
  checkSaveStatus without updating
end requireValidDocumentForCtags

to findTagFile()
  set findTagfileScript to (join of {UnixPath, "climb", "-b \"$(dirname", quoted form of documentPath(), ")\"", "tags"} by space)
  try
    do shell script findTagfileScript
  on error
    error "Unable to locate tags file"
  end try
end findTagFile

Getting the candidate tag is harder than getting the path to the tag file, mostly because it is not as well-defined of a task. Since Ctags can index lots of different languages, it won't be easy to get a solution that is right for every language. Instead, I'll define a handler that works reasonably for a lot of languages, and maintains the possibility for the user to specify the candidate precisely. This latter case is straightforward: if there is text selected in SEE, we'll search for that tag.

When no text is selected, we need to get a candidate tag in some other way. To me, it makes sense that finding symbol definitions should let the user give a term in a dialog, and that text completion should work by using the text preceding the cursor. But how much text should be used? I don't think that the longest possible tag makes sense, as that would mean, e.g., a method invocation in Python of form obj.method would use the whole thing, even though that full term is unlikely to be indexed in the tag file. Instead, it would be better to just use method as the candidate tag. A reasonable choice for many languages would then be to take the longest string of alphanumeric characters and underscores, right to left from the insertion point. Those choices lead to the handler:

to determineSearchTerm given userIntervention:shouldAsk
  set {startChar, nextChar} to selectionRange without extendingFront and extendingEnd
  if startChar is equal to nextChar then
    -- empty selection
    if shouldAsk then
      try
        display dialog "Enter search term:" default answer "" with title "Find Definition"
      on error number -128
        error "User canceled" number 901
      end try
      text returned of result
    else
      -- try the whole line
      set selectionContents to extendedSelectionText with extendingFront without extendingEnd
      get shellTransform of the selectionContents for "" thru "sed -E -e 's/.*([[:<:]][[:alnum:]_]+)$/\\1/'" without alteringLineEndings
      -- sed returns lines that are terminated with linefeeds, so get text before the final linefeed
      paragraph -2 of the result
    end if
  else
    -- just use the selection; there is too much variation in what could be a tag to guess
    selectionText()
  end if
end determineSearchTerm

The handlers presented in this post are enough to get the path to the tag file and a (partial) tag to search for. Next time, I'll connect these values from SubEthaEdit to the shell scripts handling the lookup.

Tag Matching

Our goal remains to add support for Ctags to an application. We know how to locate the relevant tags file, but what do we do with it? Fundamentally, we use the tag file to match identifiers against tags indexed by Ctags; let's make that specific, restricting ourselves for the moment to just working in the shell.

The tag file is structured as sorted lines of tab-separated records. The first field in the line is the tag, other fields identify the position of the tag in a particular source file. With this, we can check a candidate tag $TAG against the tag file $TAGFILE using look:

look "$TAG" "$TAGFILE"

Easy and fast.

To use tags to find the definition of a symbol, we'll want to hang onto all the information about each matching tag; the above use of look is all we need. For use in text completion, we'll want a longer pipeline eliminating extraneous information:

look "$TAG" "$TAGFILE" | cut -f1 | sort -u

The pipeline drops all fields but the first, the tag field, using cut and eliminates duplicates with sort -u (I suspect that uniq should work here, but look is curiously unspecific about whether it always produces its output in sorted order).

And that's it for matching tags. The file format was clearly set up with just this sort of use in mind. More details on the file format are available elsewhere.

Find that Tags File!

Our first challenge in incorporating Ctags into an editor is locating the tags file. A first attempt might be to look for a file named tags in the same directory as the document in the frontmost editor window. But this isn't quite good enough. Ctags can create a tags file by recursively descending into subdirectories, so a useful tags file might be located somewhere higher in the directory tree.

It seems like there should be a standard shell command to search upward in the directory tree, but I couldn't find it. The task isn't really that hard, so I wrote a shell script climb to do it instead of spending more time fruitlessly searching. Usage is patterned after which. To look for a tags file that recursively indexed the present directory, just do climb tags. Options are available to set where the search starts and stops.

Here's my script:

#!/bin/sh
#
# climb -- locate a file by ascending the directory tree
#
# climb [-b bottomdir] [-t topdir] filename
#
# Climb directory tree looking for a file named filename. The search
# starts by checking in the bottom directory (defaults to the current
# directory), with each parent directory checked until either the
# file is found or the top directory (defaults to root) is reached.
#


# Options allow setting the search range. Defaults are starting the
# search in the current directory and ending at root.
upTo="/"
upFrom="$PWD"

while getopts b:t: opt
do
    case $opt in
    b)  upFrom="$OPTARG"
        if ! [ -d "$upFrom" ]
        then
            echo $0: $upFrom: No such directory >&2
            exit 2
		else
			# standardize the lowermost directory path
        	upFrom="$(cd "$upFrom" && pwd -P)"
        fi
        ;;
    t)  upTo="$OPTARG"
        if ! [ -d "$upTo" ]
        then
            echo $0: $upTo: No such directory >&2
            exit 2
		else
			# standardize the uppermost directory path
        	upTo="$(cd "$upTo" && pwd -P)"
        fi
        ;;
    esac
done
shift $((OPTIND - 1))

targetFile="$1"

# To ensure termination, require that the uppermost directory is
# an ancestor of the directory where the search begins.
indx=$(awk -v d1="$upTo" -v d2="$upFrom" 'BEGIN { print index(d2, d1) }')
if ! [ $indx -eq 1 ]
then
    echo $0: $upFrom is not a descendant of $upTo >&2
fi

# Check each directory for the target file, moving up the directory tree
# until either the target is found or the uppermost directory has been
# searched. Both the lowermost directory and the uppermost directory
# are checked for the file.
while true
do
    if [ -f "$upFrom/$targetFile" ]
    then
        break
    fi
    if [ "X$upTo" = "X$upFrom" ] || [ -z "$upFrom" ] || [ "X$upFrom" = "X/" ]
    then
        exit 1
    else
        upFrom=$(dirname "$upFrom")
    fi
done

echo "$upFrom/$targetFile"

Most of the script deals with establishing the starting and ending points of the search, which I referred to in the script as the bottommost and topmost directories, respectively. They're put into a standardized format and tested for consistency, then used to define the search. The search is simple, amounting to nothing more than successively chopping off the last element of the directory path and seeing if the target file is in the resulting directory. The search stops when the topmost directory is reached, or when root is reached, just in case.

The script is general purpose, suitable for finding more than just tags files. I have mostly just called climb from AppleScripts in SubEthaEdit, with a pretty well-behaved file name and start directory. It may well be that more complex use would reveal bugs, so use with caution.

Exploring Ctags: Motivations

I've been vaguely aware of Ctags for years, but only in the last few months have I gotten a handle on how it would benefit me. Part of the problem is that most mentions of Ctags seem to assume you already know the benefits: the Wikipedia entry does this, as does the Exuberant Ctags site. Worse, many discussions make it seem that it is just an auxiliary for vi-family editors, so perhaps not even relevant to those who, like me, haven't seriously used a vi derivative in years.

After seeing an explanation in the context of BBEdit, I have a much better idea of what Ctags provides. Essentially, it generates an index called a tags file that allows for easier code navigation across multiple files, in particular providing text completions and navigating to the definition of functions or other symbols. Within BBEdit, tags also are used to improve syntax highlighting.

I must admit that I find some of the praise for it to be overblown, but maybe I just need to try it. Of course, I don't use BBEdit, either. In fact, no editor that I regularly use supports Ctags. Let's do something about that. I'll work in the context of SubEthaEdit (SEE), since I have a fair amount of experience with scripting it, and of Exuberant Ctags, since it supports more languages than the Ctags built into Mac OS X.

I'll add two features to SEE, text completion and finding definitions. To some extent, these are redundant, in that SEE has text completions and a function pop-up, but they don't extend across multiple files in the same way as Ctags. I won't be able to do anything with syntax highlighting, as in BBEdit, but it should still be enough to try out Ctags.

Both features will be structured as AppleScripts invoking shell scripts to do most of the work. The AppleScripts both have a similar structure, consisting of:

1. locating the tags file


2. determining a search term to match against the tags file


3. identifying and processing matching tags


4. letting the user select from the matching tags


5. doing something with the selection

I'll break these stages out into several posts.

DWM AppleScripts

I've been experimenting with new time management systems from Mark Forster, first trying Autofocus v. 4 and now DWM. I've found AF4 to be quite nice over the last few weeks, and like what I've seen of DWM over the last few days. In each case, I've used iCal to manage the tasks in the system.

With DWM, I keep my at-home tasks as iCal todos on a separate calendar (my work tasks are still in AF4, but will be switched over soon). Each todo has a due date; the due date here doesn't mean "do on this date," but instead means "do by this date." I keep the tasks sorted by due date. For tasks that really must be done on a particular date, put them on a different calendar, and they'll appear at the top of the list on that due date. This works well, but it is a little annoying to regularly set the due dates by hand.

The todos are set with a regular pattern, to either the next week or the next month. This is scriptable. Here is the next-week script, which I saved as "To Do Within 7 Days" under the iCal application scripts:

setDueDate of (7*days) for selectedToDo()

to setDueDate of timeFrame for task
	set newDate to (current date) + timeFrame
	tell application "iCal" to set due date of task to newDate
end setDueDate

on selectedToDo()
	set referenceText to iCalSelectionText at 1
	tell application "iCal"
		repeat with cal in calendars
			set matches to (todos of cal where summary is equal to referenceText)
			if (count of matches) > 0 then
				exit repeat
			end if
		end repeat
		if (count of matches) is equal to 0 then
			error "No matching to-do item found."
		end if
		first item of matches
	end tell
end selectedToDo

on iCalSelectionText at timeDelay
	set the oldClipboard to the clipboard
	try
		copyICalSelection at timeDelay
		set selectionText to the clipboard
	on error errText number errNum
		set the clipboard to the oldClipboard
		error errText number errNum
	end try
	set the clipboard to the oldClipboard
	selectionText
end iCalSelectionText

on copyICalSelection at timeDelay
	tell application "iCal" to activate
	tell application "System Events"
		tell process "iCal"
			keystroke return
			keystroke "c" using {command down}
			keystroke return
		end tell
	end tell
	delay timeDelay
end copyICalSelection

The next-month script is similar, just replace the 7*days by 30*days.

The bulk of the script, and the only thing tricky about it, is getting a selected to-do item; the iCal scripting dictionary provides no way to do this! The handlers selectedToDo, iCalSelectionText, and copyICalSelection are a work around. I didn't come up with this approach, it comes from a Mac OS X Hints contributor.

Overall, I'm liking DWM a lot, but I doubt I'd like it without the scripts. Because of the nature of the system, I'll make no recommendation either for or against using DWM until a month has passed, but I already do think it is quite interesting and worth taking a look at.

Update: You can download compiled scripts here.

Replacement for SubEthaEdit's Command Line Tool

The see command line tool for SubEthaEdit makes scripting needlessly complex. Because it tries to write the contents of the document to stdout upon close of the document, you wind up having to jump through hoops to get sensible behavior. The end result is that it is easy to write an AppleScript where SubEthaEdit calls out to the shell, but hard to have the shell communicate back to SubEthaEdit.

Here, I present an alternative. It is a shell script that uses osascript to open a document in SubEthaEdit. Optionally, a specific line can be given, and, if UI scripting is enabled, the document will be scrolled to show the line.

#!/bin/sh

PROGRAM=$(basename $0)

usage()
{
    echo "Usage: $PROGRAM [-gh] filename"
}

lineGiven=false
while getopts :g:h opt
do
    case $opt in
    g)      lineGiven=true
            lineToShow="$OPTARG"
            ;;
    h)      usage
            exit 0
            ;;
    '?')    echo "$PROGRAM: invalid option -$OPTARG" >&2
            usage >&2
            exit 1
            ;;
    esac
done

shift $((OPTIND - 1))

fileName="$1"

if [ -f "$fileName" ]
then
    Dir="$(dirname "$fileName")"
    Base="$(basename "$fileName")"
    AbsDir="$(cd "$Dir" && pwd -P)"
    AbsPath="$AbsDir/$Base"
else
    echo "Unknown file: $fileName" >&2
    exit 1
fi

echo $AbsPath

/usr/bin/osascript > /dev/null <<ASCPT
    set fileToOpen to POSIX file "$AbsPath"

    if $lineGiven then
        showLine of fileToOpen at ${lineToShow:-0}
    else
        tell application "SubEthaEdit"
            activate
            open fileToOpen
        end tell
    end if

    to showLine of fileToOpen at lineNumber
        tell application "SubEthaEdit"
            activate
            set activeFile to open fileToOpen
            tell activeFile
                set selection to paragraph lineNumber
            end tell
        end tell
        scrollToVisible()
    end showLine

    to scrollToVisible()
        tell application "System Events"
            if UI elements enabled then
                tell process "SubEthaEdit"
                    tell menu bar 1
                        tell menu bar item "Find"
                            tell menu 1
                                click menu item "Jump to Selection"
                            end tell
                        end tell
                    end tell
                end tell
            end if
        end tell
    end scrollToVisible
ASCPT

The Hard Part of Getting Started With Scala

In an interesting interview, Martin Odersky notes:

Clearly it is easiest for a Java or .NET developer to learn Scala. For other communities, the stumbling blocks don't have so much to do with the language itself as with the way we package it and the way the tools are set up, which is Java specific. Once they learn how these things are set up, it should not be hard to learn the language itself.

I'd already come to this conclusion on my own, as someone who has never used Java and is trying to guess at the Java conventions needed for Scala. I think this is typical of newer languages: you can't easily learn them without knowing a specific other language already.