Code Highlighting Changes

The preceding post marks a change in how code will be highlighted on this blog, now being based on Google's client side code highlighting library. What I'm using follows what Luka Marinko describes.

Overall, I think the result is decent, but not quite as nice as what I was doing before. But it's so much easier that I'm willing to provisionally accept the new look.

Markdown in Marked script

I've lately been experimenting with writing in MultiMarkdown (MMD) format. As a whole, it's quite pleasant, but it seems best suited for documents that require some formatting, but not much formal notation. That's not the sort of writing I most often do; LaTeX remains my main writing tool.

Still, MMD is a nice option at times, and has at least one significant advantage over LaTeX: it is lightweight both in its processing and in its writing, so it is easy to do on any computer. On my Macs, I'm using SubEthaEdit (SEE) to compose and have just purchased Marked to preview. Marked is quite a nice design, intended to work with any editor (in proper Unix philosophy!) and automatically updating the preview whenever the document is saved (pleasantly reminiscent of Latexmk, albeit with a much easier task). Marked is also inexpensive (three bucks!) through the Mac App Store. Unfortunately, Marked 1.1 seems to be a little buggy, but its author has said that he's already fixed the bugs I reported, with the updated version waiting to finish the review process.

While already a pretty elegant combination, I present below a script to streamline the combination of SEE and Marked a little more. You'll need to install the Markdown mode for SEE. Show the package contents and save the script below into the Scripts folder within. While you're there, throw away the silly rot13 script that is included in the mode for some inexplicable reason. Reload modes in SEE, and you'll have a command that opens the current (Multi)Markdown document in Marked.

Amusingly, I inadvertently tried typing several things in this post using Markdown syntax (which Blogger doesn't use). It's very natural!

tell application "SubEthaEdit"
    if not (exists path of front document) then
        error "You have to save the document first"
    end if
    set docpath to the path of the front document
end tell

set mdFile to POSIX file docpath
ignoring application responses
	tell application "Marked" to open mdFile
end ignoring

on seescriptsettings()
    return {
        displayName:"Preview with Marked", 
        shortDisplayName:"Preview", 
        keyboardShortcut:"@O", 
        toolbarIcon:"ToolbarIconRun", 
        inDefaultToolbar:"yes", 
        toolbarTooltip:"Preview current document with Marked", 
        inContextMenu:"no"
    }
end seescriptsettings

Edit: Made a minor change to the script. For reasons not entirely clear to me, the original form would hang on some files (naturally, none I tested before posting about it!), with SubEthaEdit waiting for a response from Marked. The ignoring application responses takes care of that.

Modeless Scripts for SubEthaEdit

SubEthaEdit (SEE) supports mode-dependent extensions to its functionality. The mechanism for this is the embedding of AppleScripts into the mode. This lets, for example, Python documents have a Check Syntax command differently implemented from the identically named Check Syntax command for Lua documents.

All well and good, but there are tasks that are of interest across most or all modes. A prime example for a programmer's editor like SEE is commenting out lines. The basics are the same regardless of language: the line needs to begin with a specific string to indicate a comment. But the specifics do matter: Python needs hashes # for comments, Lua needs a double hyphen --, and so on.

We'd thus like to have scripts that are modeless, present in any mode, but that are customizable, appropriate to any mode. Such an AppleScript needs to go into the global scripts menu for SEE, but still allow each mode to define how the behavior of the script should be implemented.

Here's how to do it. We use an AppleScript to capture the basics of a given pattern, such as determining which lines should be commented out. The AppleScript then calls a shell script specified for the current mode.

All the components for this task have been presented previously on this blog, with code indentation exemplifying the approach. I'll use SubEthaEditTools to implement the AppleScripts. The mode-specific customization is done using a plist of environment variables. Particular tasks are done by writing an AppleScript that calls a shell command stored in an environment variable in the plist; the use of SubEthaEditTools makes these scripts quite brief.

Opening the plist is done with this script:

if not documentIsAvailable() then
	return
end

openEnvironmentSettings()

on seescriptsettings()
	return {displayName:"Customize Mode...", shortDisplayName:"Environment", inContextMenu:"no"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

The include command is not part of AppleScript, it is an m4 macro used to modularize the scripts. The logic is simple: make sure there is a document available, then use its mode to open the mode-specific environment settings.

For a particular function we include the common features and call out to the shell to do the rest. For adding or removing line-oriented comments, I used:

if not documentIsAvailable() then
	return
end

if (modeSetting for "COMMENTER") is missing value then
	display alert "Commenting not available." message "You need to define COMMENTER for the mode."
	return
end

completeSelectedLines()
set outText to shellTransform of selectionText() for modeEnvironment() through "eval $COMMENTER" without alteringLineEndings
setSelectionText to outText

-- SubEthaEdit settings

on seescriptsettings()
    {displayName:"Un/Comment Selected Lines", keyboardShortcut:"@/", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

Again, the logic is simple: get the COMMENTER shell command from the mode-specific environment for the front current document, and use it to transform the selected lines.

So what should the shell command be? One possibility is the comment script presented in the preceding post. Some examples are given in that post: use those as the value in the environment variable plist, with COMMENTER as the key. There is no default comment method, so it's necessary to provide values for each of the modes you use. In practice, this isn't bad, since you can often just copy and paste between modes with minimal or no changes.

It's just as easy to define a script for block comments, such as those used in C or SML. I'll omit the details.

The AppleScripts and some supporting scripts are available for download.

A Better Comment Script

A couple of years ago, I presented a shell script for commenting out lines, for use in a LaTeX mode for SubEthaEdit. The script is an improvement over the AppleScript approach used in other SubEthaEdit modes, but does something I don't really like: it always inserts or removes comments at the beginning of the lines, rather than at an appropriate indentation level.

Below, I present line-comment, an awk script that handles line-oriented comments, taking indentation into account. Lines to comment or uncomment are read from Standard Input, and the processed lines are written to Standard Output. The script uses the current commenting of the lines to determine whether to comment or uncomment.

There are two options that can be set. First, there is the TabWidth, which defines an indentation level; this defaults to the (basically useless) Unix standard of an eight-space tab. You'll almost always want to set this, even if you're using tabs, not spaces, for indentation. Second, there is the CommentString, whose meaning should be obvious; this defaults to the hash character # common to many programming languages.

As an example, a nice choice for Python could be line-comment TabWidth=4, while for Scala line-comment TabWidth=2 CommentString="//" would be more suitable.

Update: If the script doesn't seem to work, try setting the environment variable COMMAND_MODE=unix2003. This is relevant if you want to call it from SubEthaEdit: SEE uses COMMAND_MODE=legacy, which can cause the regular expressions to fail to match.

The script is unchanged:

#! /usr/bin/awk -f

function trimIndent(text, indRE,   n, tokens) {
  # Returns the text with the indentation removed. Sets
  # global variable IndentLevel to show how many
  # indentation levels were removed.
  n = split(text, tokens, indRE)
  if (n > 1) {
    IndentLevel = n-1
    rest = tokens[n]
  } else {
    IndentLevel = 0
    rest = text
  }
  return rest
}

function commentIndex(txt, commtxt,      n) {
  n = index(txt, commtxt)
  if (n > 0 && substr(txt, 1, n-1) !~ /^[ \t]*$/) {
    n = 0
  }
  return n
}

function noncommentPrefix(txt) {
  return match(txt, /^[ \t]*/) ? substr(txt, 1, RLENGTH) : ""
}

function multiString(str, mult,     n, mstr) {
  mstr = ""
  for (n=0; n<mult; n++) {
    mstr = mstr str
  }
  return mstr
}

function offsetString(offset) {
  return multiString(" ", offset)
}

BEGIN {
  TabWidth=8
  CommentString="#"
}

NR == 1 {
  # Establish regex based on tab settings. This comes after
  # the BEGIN to allow the TabWidth to be overriden.
  indentRegex = "( {0,"(TabWidth-1)"}\t| {"TabWidth"})"
  # indentRegex = "( {0,3}\t|"offsetString(TabWidth)")"
}

{
  # Common processing for all lines. Divide the line into a
  # prefix of whitespace, followed immediately by the
  # comment string, if present, or a non-tab, non-space
  # character if not. The prefix consists of indentation
  # steps followed by an offset, defined as a number of spaces
  # insufficient to constitute an indentation step.
  Line[NR] = $0
  commInd = commentIndex($0, CommentString)
  if (commInd > 0) {
    prefix = substr($0, 1, commInd-1)
    CommentPosition[NR] = commInd
  } else {
    prefix = noncommentPrefix($0)
  }
  offset = length(trimIndent(prefix, indentRegex))
}

NR == 1 {
  BaseOffset = offset
  BaseIndent = IndentLevel
  MinOffset = BaseOffset
  MinIndent = BaseIndent
}

NR > 1 {
  if (IndentLevel < MinIndent ) {
    MinIndent = IndentLevel
    MinOffset = 0
  } else if (offset < MinOffset) {
    MinOffset = offset
  }
}

END {
  commLen = length(CommentString)
  if (length(CommentPosition) == length(Line) && MinIndent == BaseIndent && MinOffset == BaseOffset) {
    for (n=1; n<=NR; n++) {
      commPos = CommentPosition[n]
      print substr(Line[n], 1, commPos-1) substr(Line[n], commLen+commPos)
    }
  } else {
    indentPart = MinIndent ? indentRegex"{"MinIndent"}" : ""
    # indentPart = multiString(indentRegex, MinIndent)
    offsetPart = offsetString(MinOffset)
    # offsetPart = offsetString(MinOffset)
    commRegex = "^" indentPart offsetPart
    for (n=1; n<=NR; n++) {
      match(Line[n], commRegex)
      print substr(Line[n], 1, RLENGTH) CommentString substr(Line[n], 1+RLENGTH)
    }
  }
}

TaskPaper Mode for SubEthaEdit

TaskPaper is an application for managing simple to-do lists. It is somewhere between a text editor and an outline processor, focused on lists of to-do items that can be checked off. The lists can be organized into projects and marked up with tags, enabling search and selection by tag.

TaskPaper saves its documents as plain text. They can be opened, modified, and created by any application that can work with text files. In fact, it is fair to say that TaskPaper is both an application and a lightweight text-markup system specifically for to-do lists. It is pretty easy to support the TaskPaper file format, and it has been done for several text editors.

Some months back, I created a ToDo mode for SubEthaEdit that supports the TaskPaper format. Today, I finally got around to making it available for download. The mode supports syntax highlighting and has scripts to automate creating new tasks and projects, marking tasks as done, and archiving completed tasks. Tags are detected and highlighted, but there is unfortunately no way to do the outline-processor-style hoisting of particular tags. To aid in managing multiple tasks, project names appear in the function popup menu.

It is also possible to modify how the mode handles marking tasks as completed and archiving them. This requires an additional script to open a plist of environment settings; install this script in the scripts folder for SubEthaEdit (if you're not sure where that is, use Open Scripts Folder under the scripts menu in SEE). Two relevant keys can be set, SEE_TODO_MARK_DONE and SEE_TODO_ARCHIVE_DONE. The values should be set to shell commands that implement the desired behavior for marking tasks as completed and for moving completed tasks to the Archive pseudo-project.

One possibility is to pass different command-line options to the scripts that implement the default behavior for the mode. For example, you could set the value of SEE_TODO_MARK_DONE to '"$SEE_MODE_RESOURCES"/bin/markdone.sh -c -t' and the value of SEE_TODO_ARCHIVE_DONE to '"$SEE_MODE_RESOURCES/bin/archivecompleted.awk" -v Mode=c' (including the quotes in the values). With these flags, tasks are no longer marked complete with a @done tag, but instead the leading hyphen is turned into a plus sign, giving a sort of check-off effect instead of a tagging effect. Be aware that this breaks compatibility with the TaskPaper application.

Update: The ToDo mode is available on the Coding Monkeys website.

Exploring Ctags: Summary

To facilitate learning about Ctags, I've written two AppleScripts and several supporting shell scripts. These scripts were not written by an expert on Ctags, so there may be some sub-optimal, or outright wrong, choices in how they were implemented. Please let me know of any bugs found or suggestions for possible improvements.

The AppleScripts use Ctags to add a couple of features to SubEthaEdit (SEE). First, there is the text completion AppleScript, which looks up a string in the tag file and identifies possible matches. SEE already does text completion, but only in open files; by using Ctags as a basis for completions, matching symbols can be found across all the files in a large project. The second AppleScript finds definitions of selected symbols, again facilitating working with a large number of files.

The interactions with the tag file are handled using shell scripts. These are written to handle tag files created by invoking Exuberant Ctags with a variety of different options, notably including either absolute or relative paths and either numeric or ex pattern references for the location in the files. The shell scripts need to be placed somewhere on the paths defined in the AppleScripts; if in doubt, ~/Library/Application Support/SubEthaEdit/bin/ will work.

A zip archive with the scripts is available for download.

The scripts are described in a series of blog posts:

1. Exploring Ctags: Motivations


2. Find That Tags File!


3. Tag Matching


4. Ctags in SubEthaEdit


5. Ctags from SubEthaEdit to the Shell


6. Text Completions with Ctags in SubEthaEdit


7. Finding Definitions with Ctags in SubEthaEdit

Update: I've added another AppleScript and accompanying shell script for creating or updating a tag file for the front document in SEE. These are now in the zip archive, available at the same download link given above.

Finding Definitions with Ctags in SubEthaEdit

As with using Ctags for text completion, finding definitions for symbols can be expressed largely in terms of the shell scripts and AppleScript handlers already presented. Another handler, openTaggedSources, is needed, which will open files to the location of the selected tag or tags.

The resulting AppleScript is again quite concise:

on seescriptsettings()
  {displayName:"Find Definition using Ctags", shortDisplayName:"Ctags Definition", keyboardShortcut:"@^f", inContextMenu:"yes"}
end seescriptsettings

try
  requireValidDocumentForCtags()
  set tagfilepath to findTagFile()
  set searchTerm to determineSearchTerm with userIntervention
  set taglist to (pipeMatches of searchTerm out of tagfilepath thru "")
  set tagsToOpen to (pickTags from taglist with multipleSelectionsAllowed)
  openTaggedSources for tagsToOpen from tagfilepath
on error errMsg number errNum
  if errNum is equal to 901 then
    return
  else if errNum is equal to 902 then
    beep
    return
  else
    error errMsg number errNum
  end if
end try

The structure directly parallels that used for the text completion script.

Let's take a look inside the openTaggedSources handler. My approach is to dump all the selected tags back to the shell, where the shell script open-tag-files will finish the job. Here's the handler:

to openTaggedSources for tags from tagfile
  --pass tags to external script that opens them in SEE
  set exportTagsFile to "export TAGDIR=\"$(dirname " & (quoted form of tagfile) & ")\";"
  set openTagFilesPipeline to join of {"printf " & quoted form of tags, "open-tag-files RelTo=\"$TAGDIR\""} by "|"
  set openTagFilesScript to join of {UnixPath, exportTagsFile, openTagFilesPipeline, "&> /dev/null &"} by space
  do shell script openTagFilesScript
end openTaggedSources

I pass the location of the tag file to the script, so that either absolute or relative paths can be used in the tag files. Otherwise, it's just passing the selected tags out as stdin to open-tag-files in a straightforward way.

So let's look at open-tag-files:

#! /usr/bin/awk -f

BEGIN {
    FS="\t"
}

{
    # Treat relative filenames as relative to RelTo
    if ($2 ~ /^\//) {
        filePath = $2
    } else {
        filePath = RelTo "/" $2
    }
    # Handle both numeric and regex patterns
    if ($3 ~ /^[[:digit:]]+(;\")?$/) {
        match($3, /^[[:digit:]]+/)
        gotoLine = "-g " substr($3, RSTART, RLENGTH)
    } else {
        patternPlusExtras = substr($0, index($0, $3))
        numTokens = split(patternPlusExtras, token, "/")
        if (length(token[1])) {
            # Pattern looks invalid, so can't specify the line
            gotoLine = ""
        } else {
            exQuery = ""
            for (n=2; n<=numTokens; n++) {
                exQuery = exQuery "/" token[n]
                if (token[n] !~ /[^\\](\\\\)*\\$/) {
                    break
                }
            }
            exQuery = exQuery "/"
            command = "cat '"filePath"' | sed -e '"exQuery" q' | wc -l"
            command | getline lineCount
            close(command)
            gotoLine = "-g "lineCount
        }
    }
    #printf("see %s \"%s\" &\n", gotoLine, filePath)
    system("altsee "gotoLine" \""filePath"\" &")
}

This is an awk script which mostly consists of handling different ways that the tag file can be structured. Since the point is to provide a platform for experimenting with Ctags, it seems premature to commit to specific choices of absolute or relative paths, numeric line references or ex patterns, extended fields from Exuberant Ctags or just vanilla Ctags lines. For what it is worth, I'm invoking Exuberant Ctags as ctags -n --fields=+a+m+n+S -R (but there may well be better choices).

At the end open-tag-files, I use altsee to open the source files. This is a replacement for the see command line tool that comes with SubEthaEdit. I find that see is a bit of a hassle for this sort of use, so gave up on it for here (if you can get open-tag-files to work cleanly with multiple selected files, I'd love to hear about how!).

All the scripts and handlers need to be assembled into a compiled AppleScript in ~/Library/Application Support/SubEthaEdit/Scripts/ with the shell scripts set to be executable and on the path defined in the AppleScripts. If you're not sure where to put the shell scripts, I'd suggest creating a ~/Library/Application Support/SubEthaEdit/bin/ directory for SubEthaEdit-related shell scripts, and putting the scripts there. A compiled script with the needed shell script support is available for download.