Applied Abstraction

Code Highlighting Changes

2011-07-20T11:53:00.001-07:00

The preceding post marks a change in how code will be highlighted on this blog, now being based on Google's client side code highlighting library. What I'm using follows what Luka Marinko describes.

Overall, I think the result is decent, but not quite as nice as what I was doing before. But it's so much easier that I'm willing to provisionally accept the new look.

Markdown in Marked script

2011-07-20T11:38:00.001-07:00

I've lately been experimenting with writing in MultiMarkdown (MMD) format. As a whole, it's quite pleasant, but it seems best suited for documents that require some formatting, but not much formal notation. That's not the sort of writing I most often do; LaTeX remains my main writing tool.

Still, MMD is a nice option at times, and has at least one significant advantage over LaTeX: it is lightweight both in its processing and in its writing, so it is easy to do on any computer. On my Macs, I'm using SubEthaEdit (SEE) to compose and have just purchased Marked to preview. Marked is quite a nice design, intended to work with any editor (in proper Unix philosophy!) and automatically updating the preview whenever the document is saved (pleasantly reminiscent of Latexmk, albeit with a much easier task). Marked is also inexpensive (three bucks!) through the Mac App Store. Unfortunately, Marked 1.1 seems to be a little buggy, but its author has said that he's already fixed the bugs I reported, with the updated version waiting to finish the review process.

While already a pretty elegant combination, I present below a script to streamline the combination of SEE and Marked a little more. You'll need to install the Markdown mode for SEE. Show the package contents and save the script below into the Scripts folder within. While you're there, throw away the silly rot13 script that is included in the mode for some inexplicable reason. Reload modes in SEE, and you'll have a command that opens the current (Multi)Markdown document in Marked.

Amusingly, I inadvertently tried typing several things in this post using Markdown syntax (which Blogger doesn't use). It's very natural!


tell application "SubEthaEdit"
    if not (exists path of front document) then
        error "You have to save the document first"
    end if
    set docpath to the path of the front document
end tell

set mdFile to POSIX file docpath
ignoring application responses
	tell application "Marked" to open mdFile
end ignoring

on seescriptsettings()
    return {
        displayName:"Preview with Marked", 
        shortDisplayName:"Preview", 
        keyboardShortcut:"@O", 
        toolbarIcon:"ToolbarIconRun", 
        inDefaultToolbar:"yes", 
        toolbarTooltip:"Preview current document with Marked", 
        inContextMenu:"no"
    }
end seescriptsettings

Edit: Made a minor change to the script. For reasons not entirely clear to me, the original form would hang on some files (naturally, none I tested before posting about it!), with SubEthaEdit waiting for a response from Marked. The ignoring application responses takes care of that.

Modeless Scripts for SubEthaEdit

2010-03-06T10:26:00.001-08:00

SubEthaEdit (SEE) supports mode-dependent extensions to its functionality. The mechanism for this is the embedding of AppleScripts into the mode. This lets, for example, Python documents have a Check Syntax command differently implemented from the identically named Check Syntax command for Lua documents.

All well and good, but there are tasks that are of interest across most or all modes. A prime example for a programmer's editor like SEE is commenting out lines. The basics are the same regardless of language: the line needs to begin with a specific string to indicate a comment. But the specifics do matter: Python needs hashes # for comments, Lua needs a double hyphen --, and so on.

We'd thus like to have scripts that are modeless, present in any mode, but that are customizable, appropriate to any mode. Such an AppleScript needs to go into the global scripts menu for SEE, but still allow each mode to define how the behavior of the script should be implemented.

Here's how to do it. We use an AppleScript to capture the basics of a given pattern, such as determining which lines should be commented out. The AppleScript then calls a shell script specified for the current mode.

All the components for this task have been presented previously on this blog, with code indentation exemplifying the approach. I'll use SubEthaEditTools to implement the AppleScripts. The mode-specific customization is done using a plist of environment variables. Particular tasks are done by writing an AppleScript that calls a shell command stored in an environment variable in the plist; the use of SubEthaEditTools makes these scripts quite brief.

Opening the plist is done with this script:

if not documentIsAvailable() then
	return
end

openEnvironmentSettings()

on seescriptsettings()
	return {displayName:"Customize Mode...", shortDisplayName:"Environment", inContextMenu:"no"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

The include command is not part of AppleScript, it is an m4 macro used to modularize the scripts. The logic is simple: make sure there is a document available, then use its mode to open the mode-specific environment settings.

For a particular function we include the common features and call out to the shell to do the rest. For adding or removing line-oriented comments, I used:

if not documentIsAvailable() then
	return
end

if (modeSetting for "COMMENTER") is missing value then
	display alert "Commenting not available." message "You need to define COMMENTER for the mode."
	return
end

completeSelectedLines()
set outText to shellTransform of selectionText() for modeEnvironment() through "eval $COMMENTER" without alteringLineEndings
setSelectionText to outText

-- SubEthaEdit settings

on seescriptsettings()
    {displayName:"Un/Comment Selected Lines", keyboardShortcut:"@/", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

Again, the logic is simple: get the COMMENTER shell command from the mode-specific environment for the front current document, and use it to transform the selected lines.

So what should the shell command be? One possibility is the comment script presented in the preceding post. Some examples are given in that post: use those as the value in the environment variable plist, with COMMENTER as the key. There is no default comment method, so it's necessary to provide values for each of the modes you use. In practice, this isn't bad, since you can often just copy and paste between modes with minimal or no changes.

It's just as easy to define a script for block comments, such as those used in C or SML. I'll omit the details.

The AppleScripts and some supporting scripts are available for download.

A Better Comment Script

2010-02-28T04:41:00.001-08:00

A couple of years ago, I presented a shell script for commenting out lines, for use in a LaTeX mode for SubEthaEdit. The script is an improvement over the AppleScript approach used in other SubEthaEdit modes, but does something I don't really like: it always inserts or removes comments at the beginning of the lines, rather than at an appropriate indentation level.

Below, I present line-comment, an awk script that handles line-oriented comments, taking indentation into account. Lines to comment or uncomment are read from Standard Input, and the processed lines are written to Standard Output. The script uses the current commenting of the lines to determine whether to comment or uncomment.

There are two options that can be set. First, there is the TabWidth, which defines an indentation level; this defaults to the (basically useless) Unix standard of an eight-space tab. You'll almost always want to set this, even if you're using tabs, not spaces, for indentation. Second, there is the CommentString, whose meaning should be obvious; this defaults to the hash character # common to many programming languages.

As an example, a nice choice for Python could be line-comment TabWidth=4, while for Scala line-comment TabWidth=2 CommentString="//" would be more suitable.

Update: If the script doesn't seem to work, try setting the environment variable COMMAND_MODE=unix2003. This is relevant if you want to call it from SubEthaEdit: SEE uses COMMAND_MODE=legacy, which can cause the regular expressions to fail to match.

The script is unchanged:

#! /usr/bin/awk -f

function trimIndent(text, indRE,   n, tokens) {
  # Returns the text with the indentation removed. Sets
  # global variable IndentLevel to show how many
  # indentation levels were removed.
  n = split(text, tokens, indRE)
  if (n > 1) {
    IndentLevel = n-1
    rest = tokens[n]
  } else {
    IndentLevel = 0
    rest = text
  }
  return rest
}

function commentIndex(txt, commtxt,      n) {
  n = index(txt, commtxt)
  if (n > 0 && substr(txt, 1, n-1) !~ /^[ \t]*$/) {
    n = 0
  }
  return n
}

function noncommentPrefix(txt) {
  return match(txt, /^[ \t]*/) ? substr(txt, 1, RLENGTH) : ""
}

function multiString(str, mult,     n, mstr) {
  mstr = ""
  for (n=0; n<mult; n++) {
    mstr = mstr str
  }
  return mstr
}

function offsetString(offset) {
  return multiString(" ", offset)
}

BEGIN {
  TabWidth=8
  CommentString="#"
}

NR == 1 {
  # Establish regex based on tab settings. This comes after
  # the BEGIN to allow the TabWidth to be overriden.
  indentRegex = "( {0,"(TabWidth-1)"}\t| {"TabWidth"})"
  # indentRegex = "( {0,3}\t|"offsetString(TabWidth)")"
}

{
  # Common processing for all lines. Divide the line into a
  # prefix of whitespace, followed immediately by the
  # comment string, if present, or a non-tab, non-space
  # character if not. The prefix consists of indentation
  # steps followed by an offset, defined as a number of spaces
  # insufficient to constitute an indentation step.
  Line[NR] = $0
  commInd = commentIndex($0, CommentString)
  if (commInd > 0) {
    prefix = substr($0, 1, commInd-1)
    CommentPosition[NR] = commInd
  } else {
    prefix = noncommentPrefix($0)
  }
  offset = length(trimIndent(prefix, indentRegex))
}

NR == 1 {
  BaseOffset = offset
  BaseIndent = IndentLevel
  MinOffset = BaseOffset
  MinIndent = BaseIndent
}

NR > 1 {
  if (IndentLevel < MinIndent ) {
    MinIndent = IndentLevel
    MinOffset = 0
  } else if (offset < MinOffset) {
    MinOffset = offset
  }
}

END {
  commLen = length(CommentString)
  if (length(CommentPosition) == length(Line) && MinIndent == BaseIndent && MinOffset == BaseOffset) {
    for (n=1; n<=NR; n++) {
      commPos = CommentPosition[n]
      print substr(Line[n], 1, commPos-1) substr(Line[n], commLen+commPos)
    }
  } else {
    indentPart = MinIndent ? indentRegex"{"MinIndent"}" : ""
    # indentPart = multiString(indentRegex, MinIndent)
    offsetPart = offsetString(MinOffset)
    # offsetPart = offsetString(MinOffset)
    commRegex = "^" indentPart offsetPart
    for (n=1; n<=NR; n++) {
      match(Line[n], commRegex)
      print substr(Line[n], 1, RLENGTH) CommentString substr(Line[n], 1+RLENGTH)
    }
  }
}

TaskPaper Mode for SubEthaEdit

2010-02-27T08:17:00.001-08:00

TaskPaper is an application for managing simple to-do lists. It is somewhere between a text editor and an outline processor, focused on lists of to-do items that can be checked off. The lists can be organized into projects and marked up with tags, enabling search and selection by tag.

TaskPaper saves its documents as plain text. They can be opened, modified, and created by any application that can work with text files. In fact, it is fair to say that TaskPaper is both an application and a lightweight text-markup system specifically for to-do lists. It is pretty easy to support the TaskPaper file format, and it has been done for several text editors.

Some months back, I created a ToDo mode for SubEthaEdit that supports the TaskPaper format. Today, I finally got around to making it available for download. The mode supports syntax highlighting and has scripts to automate creating new tasks and projects, marking tasks as done, and archiving completed tasks. Tags are detected and highlighted, but there is unfortunately no way to do the outline-processor-style hoisting of particular tags. To aid in managing multiple tasks, project names appear in the function popup menu.

It is also possible to modify how the mode handles marking tasks as completed and archiving them. This requires an additional script to open a plist of environment settings; install this script in the scripts folder for SubEthaEdit (if you're not sure where that is, use Open Scripts Folder under the scripts menu in SEE). Two relevant keys can be set, SEE_TODO_MARK_DONE and SEE_TODO_ARCHIVE_DONE. The values should be set to shell commands that implement the desired behavior for marking tasks as completed and for moving completed tasks to the Archive pseudo-project.

One possibility is to pass different command-line options to the scripts that implement the default behavior for the mode. For example, you could set the value of SEE_TODO_MARK_DONE to '"$SEE_MODE_RESOURCES"/bin/markdone.sh -c -t' and the value of SEE_TODO_ARCHIVE_DONE to '"$SEE_MODE_RESOURCES/bin/archivecompleted.awk" -v Mode=c' (including the quotes in the values). With these flags, tasks are no longer marked complete with a @done tag, but instead the leading hyphen is turned into a plus sign, giving a sort of check-off effect instead of a tagging effect. Be aware that this breaks compatibility with the TaskPaper application.

Update: The ToDo mode is available on the Coding Monkeys website.

Exploring Ctags: Summary

2010-02-20T11:14:00.001-08:00

To facilitate learning about Ctags, I've written two AppleScripts and several supporting shell scripts. These scripts were not written by an expert on Ctags, so there may be some sub-optimal, or outright wrong, choices in how they were implemented. Please let me know of any bugs found or suggestions for possible improvements.

The AppleScripts use Ctags to add a couple of features to SubEthaEdit (SEE). First, there is the text completion AppleScript, which looks up a string in the tag file and identifies possible matches. SEE already does text completion, but only in open files; by using Ctags as a basis for completions, matching symbols can be found across all the files in a large project. The second AppleScript finds definitions of selected symbols, again facilitating working with a large number of files.

The interactions with the tag file are handled using shell scripts. These are written to handle tag files created by invoking Exuberant Ctags with a variety of different options, notably including either absolute or relative paths and either numeric or ex pattern references for the location in the files. The shell scripts need to be placed somewhere on the paths defined in the AppleScripts; if in doubt, ~/Library/Application Support/SubEthaEdit/bin/ will work.

A zip archive with the scripts is available for download.

The scripts are described in a series of blog posts:

Exploring Ctags: Motivations

Find That Tags File!

Tag Matching

Ctags in SubEthaEdit

Ctags from SubEthaEdit to the Shell

Text Completions with Ctags in SubEthaEdit

Finding Definitions with Ctags in SubEthaEdit

Update: I've added another AppleScript and accompanying shell script for creating or updating a tag file for the front document in SEE. These are now in the zip archive, available at the same download link given above.

Finding Definitions with Ctags in SubEthaEdit

2010-02-20T10:38:00.001-08:00

As with using Ctags for text completion, finding definitions for symbols can be expressed largely in terms of the shell scripts and AppleScript handlers already presented. Another handler, openTaggedSources, is needed, which will open files to the location of the selected tag or tags.

The resulting AppleScript is again quite concise:

on seescriptsettings()
  {displayName:"Find Definition using Ctags", shortDisplayName:"Ctags Definition", keyboardShortcut:"@^f", inContextMenu:"yes"}
end seescriptsettings

try
  requireValidDocumentForCtags()
  set tagfilepath to findTagFile()
  set searchTerm to determineSearchTerm with userIntervention
  set taglist to (pipeMatches of searchTerm out of tagfilepath thru "")
  set tagsToOpen to (pickTags from taglist with multipleSelectionsAllowed)
  openTaggedSources for tagsToOpen from tagfilepath
on error errMsg number errNum
  if errNum is equal to 901 then
    return
  else if errNum is equal to 902 then
    beep
    return
  else
    error errMsg number errNum
  end if
end try

The structure directly parallels that used for the text completion script.

Let's take a look inside the openTaggedSources handler. My approach is to dump all the selected tags back to the shell, where the shell script open-tag-files will finish the job. Here's the handler:

to openTaggedSources for tags from tagfile
  --pass tags to external script that opens them in SEE
  set exportTagsFile to "export TAGDIR=\"$(dirname " & (quoted form of tagfile) & ")\";"
  set openTagFilesPipeline to join of {"printf " & quoted form of tags, "open-tag-files RelTo=\"$TAGDIR\""} by "|"
  set openTagFilesScript to join of {UnixPath, exportTagsFile, openTagFilesPipeline, "&> /dev/null &"} by space
  do shell script openTagFilesScript
end openTaggedSources

I pass the location of the tag file to the script, so that either absolute or relative paths can be used in the tag files. Otherwise, it's just passing the selected tags out as stdin to open-tag-files in a straightforward way.

So let's look at open-tag-files:

#! /usr/bin/awk -f

BEGIN {
    FS="\t"
}

{
    # Treat relative filenames as relative to RelTo
    if ($2 ~ /^\//) {
        filePath = $2
    } else {
        filePath = RelTo "/" $2
    }
    # Handle both numeric and regex patterns
    if ($3 ~ /^[[:digit:]]+(;\")?$/) {
        match($3, /^[[:digit:]]+/)
        gotoLine = "-g " substr($3, RSTART, RLENGTH)
    } else {
        patternPlusExtras = substr($0, index($0, $3))
        numTokens = split(patternPlusExtras, token, "/")
        if (length(token[1])) {
            # Pattern looks invalid, so can't specify the line
            gotoLine = ""
        } else {
            exQuery = ""
            for (n=2; n<=numTokens; n++) {
                exQuery = exQuery "/" token[n]
                if (token[n] !~ /[^\\](\\\\)*\\$/) {
                    break
                }
            }
            exQuery = exQuery "/"
            command = "cat '"filePath"' | sed -e '"exQuery" q' | wc -l"
            command | getline lineCount
            close(command)
            gotoLine = "-g "lineCount
        }
    }
    #printf("see %s \"%s\" &\n", gotoLine, filePath)
    system("altsee "gotoLine" \""filePath"\" &")
}

This is an awk script which mostly consists of handling different ways that the tag file can be structured. Since the point is to provide a platform for experimenting with Ctags, it seems premature to commit to specific choices of absolute or relative paths, numeric line references or ex patterns, extended fields from Exuberant Ctags or just vanilla Ctags lines. For what it is worth, I'm invoking Exuberant Ctags as ctags -n --fields=+a+m+n+S -R (but there may well be better choices).

At the end open-tag-files, I use altsee to open the source files. This is a replacement for the see command line tool that comes with SubEthaEdit. I find that see is a bit of a hassle for this sort of use, so gave up on it for here (if you can get open-tag-files to work cleanly with multiple selected files, I'd love to hear about how!).

All the scripts and handlers need to be assembled into a compiled AppleScript in ~/Library/Application Support/SubEthaEdit/Scripts/ with the shell scripts set to be executable and on the path defined in the AppleScripts. If you're not sure where to put the shell scripts, I'd suggest creating a ~/Library/Application Support/SubEthaEdit/bin/ directory for SubEthaEdit-related shell scripts, and putting the scripts there. A compiled script with the needed shell script support is available for download.

Text Completions with Ctags in SubEthaEdit

2010-02-20T09:12:00.001-08:00

With the infrastructure set up in the last few posts, it is now relatively easy to add Ctags-based text completions to SubEthaEdit (SEE). We use the shell scripts and AppleScript handlers to locate the tag file, determine a search term, get a list of tags matching the search term, and put up a dialog to have the user pick a tag. The only thing we're missing is a handler to actually insert the selected tag.

Here's a handler that does the job:

to insertCompletion of baseText by completionText
  -- assumes that the baseText is what was determined from the selection
  set {startChar, nextChar} to selectionRange without extendingFront and extendingEnd
  if the completionText does not start with the baseText then
    error "Invalid completion"
  end if
  if length of baseText is equal to length of completionText then
    -- completion is the same as the existing text, just position the insertion point
    setSelectionRange to nextChar
  else if startChar is equal to nextChar then
    -- empty selection, search term was inferred and only the difference needs to be included
    set completion to characters (1 + (length of baseText)) through (length of completionText) of completionText as text
    setSelectionText to completion
    setSelectionRange to nextChar + (length of completion)
  else
    --text selected, just replace it
    setSelectionText to completionText
    setSelectionRange to startChar + (length of completionText)
  end if
end insertCompletion

The handler has parameters corresponding to the base text sought for in the tag file and to the selected tag. These two strings are used, along with the length of the selection in SEE, to determine exactly how much text to insert. It would have been possible to just use the SEE selection, without passing in the base text, but it would have required essentially repeating the entire process of determining the search term; I think the design could be improved here, but I can live with this for now.

Using all these handlers, the logic for the text completion script is now expressible in a compact form:

try
  requireValidDocumentForCtags()
  set tagfilepath to findTagFile()
  set searchTerm to determineSearchTerm without userIntervention
  --set taglist to (pipeMatches of searchTerm out of tagfilepath thru "awk -F\"\\t\" '{ print $1 }' | sort -u")
  set taglist to (pipeMatches of searchTerm out of tagfilepath thru "cut -f1 | sort -u")
  set selectedTag to (pickTags from taglist without multipleSelectionsAllowed)
  insertCompletion of searchTerm by selectedTag
on error errMsg number errNum
  if errNum is equal to 901 then
    return
  else if errNum is equal to 902 then
    beep
    return
  else
    error errMsg number errNum
  end if
end try

The try block catches the errors we defined, letting any others go through for SEE to inform us about.

The last component needed is a seescriptsettings handler. I used this:

on seescriptsettings()
  {displayName:"Complete using Ctags", shortDisplayName:"Ctags Completion", keyboardShortcut:"@^t", inContextMenu:"yes"}
end seescriptsettings

All this needs to be assembled into a script, which is saved as a compiled script in ~/Library/Application Support/SubEthaEdit/Scripts/. A compiled script is available for download.

Ctags from SubEthaEdit to the Shell

2010-02-18T13:00:00.001-08:00

In the last few posts on Ctags, I've presented shell scripts for locating a tag file and looking up a tag in it, and AppleScripts for identifying what tag file should be used and what tag to search for in it. In this post, I'll present AppleScript handlers that bridge between these two scripting systems. As in the previous post, I'll use my SubEthaEditTools to simplify the process.

Essentially, the handler will need to construct a shell command that invokes look to find a tag in the tag file. Beyond that, I'll include the option to post-process the matching lines, which I'll use for text completion. For finding the definition of a tag, no post-processing is needed, so the handler checks for an empty pipeline and handles it cleanly.

The handler is:

to pipeMatches of tag out of tagfile thru pipeline
  ignoring white space
    if "" is equal to pipeline then
      set postProcess to ""
    else
      set postProcess to "| " & pipeline
    end if
  end ignoring
  set lookupScript to (join of {UnixPath, "look ", tag, quoted form of tagfile, postProcess} by space)
  try
    do shell script lookupScript
  on error
    error "Pipeline failed to process tag matches" number 902
  end try
  paragraphs of the result
end pipeMatches

Note that the handler ends by taking the paragraphs of the shell script result. This converts the lines selected by look (and any post-processing) into a list of matches.

With the two use cases in mind, the user will need to pick a relevant tag or tags from the list of matches. With text completion, only one selection makes sense, but more than one might be OK for finding definitions. Here's a handler for the two cases:

to pickTags from taglist given multipleSelectionsAllowed:allowMultiple
  try
    if allowMultiple then
      choose from list taglist with title "Matching tags" with prompt "Select tag:" default items (first item of taglist) with multiple selections allowed
      join of result by "\n"
    else
      choose from list taglist with title "Matching tags" with prompt "Select tag:" default items (first item of taglist)
      first item of the result
    end if
  on error
    -- user canceled, do nothing
    error number 901
  end try
end pickTags

We're nearly done. What remains is to assemble all these handlers into AppleScripts for the two use cases, adding whatever specifics are needed for the two tasks.

Ctags in SubEthaEdit

2010-02-17T11:38:00.001-08:00

We've now looked at how to locate the right tags file and match a tag against it by working in the shell. But our goal is to connect Ctags to an editor, SubEthaEdit (SEE) in this case. We thus will need to switch from the world of the shell to the world of AppleScript. In this post, I'll just focus on getting the path to the tags file and a tag for which to search from SEE.

I'll not be working directly with SubEthaEdit's AppleScript dictionary, instead using my SubEthaEditTools handlers as a basis. Should anyone be interested in connecting Ctags to another Mac OS X editor that supports AppleScript, it would probably be better to port the SubEthaEditTools handlers to work with the editor and directly use the scripts I'll present here.

As a general design strategy, I'll identify two AppleScript error numbers with expected behaviors. First, I'll use number 901 to indicate that tag processing should be abandoned. Second, I'll use number 902 to indicate that an error of known type has occurred. This lets me handle a broad class of troubles by either quietly exiting, or beeping then exiting. Any other errors will just be unhandled, causing SubEthaEdit to show a sheet with details of the error.

Additionally, I'll need to define a search path for shell tools. Rather than using a customizable environment as I've done before, I'll just define one as an AppleScript property:

property UnixPath : "export PATH=\"$HOME/Library/Application Support/SubEthaEdit/bin:/Library/Application Support/SubEthaEdit/bin:$HOME/Library/bin:/usr/local/bin:/opt/local/bin:/usr/bin:/bin:/usr/local/sbin:/opt/local/sbin:/usr/sbin:/sbin\";"

To find the tags file, I first need to make sure a document is available to use as the starting point for the search. Second, I just need to call out to the shell with an appropriate command. Encapsulating these in handlers, I define:

on requireValidDocumentForCtags()
  if not documentIsAvailable() then
    error "No document open" number 902
  end if
  checkSaveStatus without updating
end requireValidDocumentForCtags

to findTagFile()
  set findTagfileScript to (join of {UnixPath, "climb", "-b \"$(dirname", quoted form of documentPath(), ")\"", "tags"} by space)
  try
    do shell script findTagfileScript
  on error
    error "Unable to locate tags file"
  end try
end findTagFile

Getting the candidate tag is harder than getting the path to the tag file, mostly because it is not as well-defined of a task. Since Ctags can index lots of different languages, it won't be easy to get a solution that is right for every language. Instead, I'll define a handler that works reasonably for a lot of languages, and maintains the possibility for the user to specify the candidate precisely. This latter case is straightforward: if there is text selected in SEE, we'll search for that tag.

When no text is selected, we need to get a candidate tag in some other way. To me, it makes sense that finding symbol definitions should let the user give a term in a dialog, and that text completion should work by using the text preceding the cursor. But how much text should be used? I don't think that the longest possible tag makes sense, as that would mean, e.g., a method invocation in Python of form obj.method would use the whole thing, even though that full term is unlikely to be indexed in the tag file. Instead, it would be better to just use method as the candidate tag. A reasonable choice for many languages would then be to take the longest string of alphanumeric characters and underscores, right to left from the insertion point. Those choices lead to the handler:

to determineSearchTerm given userIntervention:shouldAsk
  set {startChar, nextChar} to selectionRange without extendingFront and extendingEnd
  if startChar is equal to nextChar then
    -- empty selection
    if shouldAsk then
      try
        display dialog "Enter search term:" default answer "" with title "Find Definition"
      on error number -128
        error "User canceled" number 901
      end try
      text returned of result
    else
      -- try the whole line
      set selectionContents to extendedSelectionText with extendingFront without extendingEnd
      get shellTransform of the selectionContents for "" thru "sed -E -e 's/.*([[:<:]][[:alnum:]_]+)$/\\1/'" without alteringLineEndings
      -- sed returns lines that are terminated with linefeeds, so get text before the final linefeed
      paragraph -2 of the result
    end if
  else
    -- just use the selection; there is too much variation in what could be a tag to guess
    selectionText()
  end if
end determineSearchTerm

The handlers presented in this post are enough to get the path to the tag file and a (partial) tag to search for. Next time, I'll connect these values from SubEthaEdit to the shell scripts handling the lookup.

Tag Matching

2010-02-14T04:32:00.001-08:00

Our goal remains to add support for Ctags to an application. We know how to locate the relevant tags file, but what do we do with it? Fundamentally, we use the tag file to match identifiers against tags indexed by Ctags; let's make that specific, restricting ourselves for the moment to just working in the shell.

The tag file is structured as sorted lines of tab-separated records. The first field in the line is the tag, other fields identify the position of the tag in a particular source file. With this, we can check a candidate tag $TAG against the tag file $TAGFILE using look:


look "$TAG" "$TAGFILE"

Easy and fast.

To use tags to find the definition of a symbol, we'll want to hang onto all the information about each matching tag; the above use of look is all we need. For use in text completion, we'll want a longer pipeline eliminating extraneous information:


look "$TAG" "$TAGFILE" | cut -f1 | sort -u

The pipeline drops all fields but the first, the tag field, using cut and eliminates duplicates with sort -u (I suspect that uniq should work here, but look is curiously unspecific about whether it always produces its output in sorted order).

And that's it for matching tags. The file format was clearly set up with just this sort of use in mind. More details on the file format are available elsewhere.

Find that Tags File!

2010-02-14T00:20:00.001-08:00

Our first challenge in incorporating Ctags into an editor is locating the tags file. A first attempt might be to look for a file named tags in the same directory as the document in the frontmost editor window. But this isn't quite good enough. Ctags can create a tags file by recursively descending into subdirectories, so a useful tags file might be located somewhere higher in the directory tree.

It seems like there should be a standard shell command to search upward in the directory tree, but I couldn't find it. The task isn't really that hard, so I wrote a shell script climb to do it instead of spending more time fruitlessly searching. Usage is patterned after which. To look for a tags file that recursively indexed the present directory, just do climb tags. Options are available to set where the search starts and stops.

Here's my script:

#!/bin/sh
#
# climb -- locate a file by ascending the directory tree
#
# climb [-b bottomdir] [-t topdir] filename
#
# Climb directory tree looking for a file named filename. The search
# starts by checking in the bottom directory (defaults to the current
# directory), with each parent directory checked until either the
# file is found or the top directory (defaults to root) is reached.
#


# Options allow setting the search range. Defaults are starting the
# search in the current directory and ending at root.
upTo="/"
upFrom="$PWD"

while getopts b:t: opt
do
    case $opt in
    b)  upFrom="$OPTARG"
        if ! [ -d "$upFrom" ]
        then
            echo $0: $upFrom: No such directory >&2
            exit 2
		else
			# standardize the lowermost directory path
        	upFrom="$(cd "$upFrom" && pwd -P)"
        fi
        ;;
    t)  upTo="$OPTARG"
        if ! [ -d "$upTo" ]
        then
            echo $0: $upTo: No such directory >&2
            exit 2
		else
			# standardize the uppermost directory path
        	upTo="$(cd "$upTo" && pwd -P)"
        fi
        ;;
    esac
done
shift $((OPTIND - 1))

targetFile="$1"

# To ensure termination, require that the uppermost directory is
# an ancestor of the directory where the search begins.
indx=$(awk -v d1="$upTo" -v d2="$upFrom" 'BEGIN { print index(d2, d1) }')
if ! [ $indx -eq 1 ]
then
    echo $0: $upFrom is not a descendant of $upTo >&2
fi

# Check each directory for the target file, moving up the directory tree
# until either the target is found or the uppermost directory has been
# searched. Both the lowermost directory and the uppermost directory
# are checked for the file.
while true
do
    if [ -f "$upFrom/$targetFile" ]
    then
        break
    fi
    if [ "X$upTo" = "X$upFrom" ] || [ -z "$upFrom" ] || [ "X$upFrom" = "X/" ]
    then
        exit 1
    else
        upFrom=$(dirname "$upFrom")
    fi
done

echo "$upFrom/$targetFile"

Most of the script deals with establishing the starting and ending points of the search, which I referred to in the script as the bottommost and topmost directories, respectively. They're put into a standardized format and tested for consistency, then used to define the search. The search is simple, amounting to nothing more than successively chopping off the last element of the directory path and seeing if the target file is in the resulting directory. The search stops when the topmost directory is reached, or when root is reached, just in case.

The script is general purpose, suitable for finding more than just tags files. I have mostly just called climb from AppleScripts in SubEthaEdit, with a pretty well-behaved file name and start directory. It may well be that more complex use would reveal bugs, so use with caution.

Exploring Ctags: Motivations

2010-02-13T11:23:00.001-08:00

I've been vaguely aware of Ctags for years, but only in the last few months have I gotten a handle on how it would benefit me. Part of the problem is that most mentions of Ctags seem to assume you already know the benefits: the Wikipedia entry does this, as does the Exuberant Ctags site. Worse, many discussions make it seem that it is just an auxiliary for vi-family editors, so perhaps not even relevant to those who, like me, haven't seriously used a vi derivative in years.

After seeing an explanation in the context of BBEdit, I have a much better idea of what Ctags provides. Essentially, it generates an index called a tags file that allows for easier code navigation across multiple files, in particular providing text completions and navigating to the definition of functions or other symbols. Within BBEdit, tags also are used to improve syntax highlighting.

I must admit that I find some of the praise for it to be overblown, but maybe I just need to try it. Of course, I don't use BBEdit, either. In fact, no editor that I regularly use supports Ctags. Let's do something about that. I'll work in the context of SubEthaEdit (SEE), since I have a fair amount of experience with scripting it, and of Exuberant Ctags, since it supports more languages than the Ctags built into Mac OS X.

I'll add two features to SEE, text completion and finding definitions. To some extent, these are redundant, in that SEE has text completions and a function pop-up, but they don't extend across multiple files in the same way as Ctags. I won't be able to do anything with syntax highlighting, as in BBEdit, but it should still be enough to try out Ctags.

Both features will be structured as AppleScripts invoking shell scripts to do most of the work. The AppleScripts both have a similar structure, consisting of:

locating the tags file

determining a search term to match against the tags file

identifying and processing matching tags

letting the user select from the matching tags

doing something with the selection

I'll break these stages out into several posts.

DWM AppleScripts

2010-02-12T22:46:00.001-08:00

I've been experimenting with new time management systems from Mark Forster, first trying Autofocus v. 4 and now DWM. I've found AF4 to be quite nice over the last few weeks, and like what I've seen of DWM over the last few days. In each case, I've used iCal to manage the tasks in the system.

With DWM, I keep my at-home tasks as iCal todos on a separate calendar (my work tasks are still in AF4, but will be switched over soon). Each todo has a due date; the due date here doesn't mean "do on this date," but instead means "do by this date." I keep the tasks sorted by due date. For tasks that really must be done on a particular date, put them on a different calendar, and they'll appear at the top of the list on that due date. This works well, but it is a little annoying to regularly set the due dates by hand.

The todos are set with a regular pattern, to either the next week or the next month. This is scriptable. Here is the next-week script, which I saved as "To Do Within 7 Days" under the iCal application scripts:

setDueDate of (7*days) for selectedToDo()

to setDueDate of timeFrame for task
	set newDate to (current date) + timeFrame
	tell application "iCal" to set due date of task to newDate
end setDueDate

on selectedToDo()
	set referenceText to iCalSelectionText at 1
	tell application "iCal"
		repeat with cal in calendars
			set matches to (todos of cal where summary is equal to referenceText)
			if (count of matches) > 0 then
				exit repeat
			end if
		end repeat
		if (count of matches) is equal to 0 then
			error "No matching to-do item found."
		end if
		first item of matches
	end tell
end selectedToDo

on iCalSelectionText at timeDelay
	set the oldClipboard to the clipboard
	try
		copyICalSelection at timeDelay
		set selectionText to the clipboard
	on error errText number errNum
		set the clipboard to the oldClipboard
		error errText number errNum
	end try
	set the clipboard to the oldClipboard
	selectionText
end iCalSelectionText

on copyICalSelection at timeDelay
	tell application "iCal" to activate
	tell application "System Events"
		tell process "iCal"
			keystroke return
			keystroke "c" using {command down}
			keystroke return
		end tell
	end tell
	delay timeDelay
end copyICalSelection

The next-month script is similar, just replace the 7*days by 30*days.

The bulk of the script, and the only thing tricky about it, is getting a selected to-do item; the iCal scripting dictionary provides no way to do this! The handlers selectedToDo, iCalSelectionText, and copyICalSelection are a work around. I didn't come up with this approach, it comes from a Mac OS X Hints contributor.

Overall, I'm liking DWM a lot, but I doubt I'd like it without the scripts. Because of the nature of the system, I'll make no recommendation either for or against using DWM until a month has passed, but I already do think it is quite interesting and worth taking a look at.

Update: You can download compiled scripts here.

Replacement for SubEthaEdit's Command Line Tool

2009-11-06T09:49:00.001-08:00

The see command line tool for SubEthaEdit makes scripting needlessly complex. Because it tries to write the contents of the document to stdout upon close of the document, you wind up having to jump through hoops to get sensible behavior. The end result is that it is easy to write an AppleScript where SubEthaEdit calls out to the shell, but hard to have the shell communicate back to SubEthaEdit.

Here, I present an alternative. It is a shell script that uses osascript to open a document in SubEthaEdit. Optionally, a specific line can be given, and, if UI scripting is enabled, the document will be scrolled to show the line.

#!/bin/sh

PROGRAM=$(basename $0)

usage()
{
    echo "Usage: $PROGRAM [-gh] filename"
}

lineGiven=false
while getopts :g:h opt
do
    case $opt in
    g)      lineGiven=true
            lineToShow="$OPTARG"
            ;;
    h)      usage
            exit 0
            ;;
    '?')    echo "$PROGRAM: invalid option -$OPTARG" >&2
            usage >&2
            exit 1
            ;;
    esac
done

shift $((OPTIND - 1))

fileName="$1"

if [ -f "$fileName" ]
then
    Dir="$(dirname "$fileName")"
    Base="$(basename "$fileName")"
    AbsDir="$(cd "$Dir" && pwd -P)"
    AbsPath="$AbsDir/$Base"
else
    echo "Unknown file: $fileName" >&2
    exit 1
fi

echo $AbsPath

/usr/bin/osascript > /dev/null <<ASCPT
    set fileToOpen to POSIX file "$AbsPath"

    if $lineGiven then
        showLine of fileToOpen at ${lineToShow:-0}
    else
        tell application "SubEthaEdit"
            activate
            open fileToOpen
        end tell
    end if

    to showLine of fileToOpen at lineNumber
        tell application "SubEthaEdit"
            activate
            set activeFile to open fileToOpen
            tell activeFile
                set selection to paragraph lineNumber
            end tell
        end tell
        scrollToVisible()
    end showLine

    to scrollToVisible()
        tell application "System Events"
            if UI elements enabled then
                tell process "SubEthaEdit"
                    tell menu bar 1
                        tell menu bar item "Find"
                            tell menu 1
                                click menu item "Jump to Selection"
                            end tell
                        end tell
                    end tell
                end tell
            end if
        end tell
    end scrollToVisible
ASCPT

The Hard Part of Getting Started With Scala

2009-08-24T12:14:00.001-07:00

In an interesting interview, Martin Odersky notes:

Clearly it is easiest for a Java or .NET developer to learn Scala. For other communities, the stumbling blocks don't have so much to do with the language itself as with the way we package it and the way the tools are set up, which is Java specific. Once they learn how these things are set up, it should not be hard to learn the language itself.

I'd already come to this conclusion on my own, as someone who has never used Java and is trying to guess at the Java conventions needed for Scala. I think this is typical of newer languages: you can't easily learn them without knowing a specific other language already.

Hidden Setting for Plaxo

2009-05-09T14:56:00.001-07:00

I use Plaxo, mainly for backing up my address book. Plaxo also lets you connect a number of online services to your Plaxo profile and share updates with connected acquaintances, providing a sort of ad hoc online social network.

And Plaxo nags you every time you go to your profile page, asking you to update your status by entering a little text into a field. There doesn't seem to be any way to turn off the status report feature, so you also don't have any way to get it to quit nagging you. But I've found the hidden setting that lets the people linked to you know that you're not using the status reports, and, more importantly, gets Plaxo to quit nagging you.

All you have to do is enter the secret text into the status report field. The secret text reads: "is not providing status reports."

"Hello World" for Numerics

2009-05-01T08:08:00.001-07:00

I'm taking a serious look at Scala, and liking what I see very much. It struck me that I have my own version of the standard Hello World program. I recognize, of course, that getting a program to print "Hello World" is not about testing I/O, but instead seeing that you can get a program running at all. Nonetheless, the first test I always make is a different one, that tells me something vital.

I do this:

[~ 479] scala
Welcome to Scala version 2.7.3.final (Java HotSpot(TM) Client VM, Java 1.5.0_16).
Type in expressions to have them evaluated.
Type :help for more information.

scala> 1 + 2*3
res0: Int = 7

If 1+2*3 returns a value other than 7, I automatically reject the language. For completeness, I also check 2*3+1. What programming I do is normally math-intensive enough that I refuse to deal with languages that can't handle normal precedence rules.

Prefix languages like the Lisp family pass this test, as they won't return a value. I've not worked with postfix languages like Forth, but I think they'd pass, too.

Experimenting with Implementing Automator Actions

2009-04-10T08:25:00.001-07:00

I've recently experimented with implementing actions for Automator using AppleScript. Overall, it was an interesting experience, and one I'd definitely recommend trying yourself.

Creating an action is generally quite simple, consisting of three main steps:

Laying out controls in Interface Builder and establishing bindings to an NSObjectController,

Defining the properties of the action by filling out some values in plists,

Writing an AppleScript to actually implement the behavior.

I found Apple's documentation and an article by Matt Neuburg from the now-defunct O'Reilly Mac developer center. The latter article is a nice tutorial, but a little dated in spots; some caution is warranted.

As a specific example, I decided to take some of the AppleScript handlers for SubEthaEdit I already had on hand and convert them into actions. After reviewing the possibilities, I decided that there were just two actions that looked sensible: getting text from the front SubEthaEdit document and setting text in the front SubEthaEdit document. These are quite similar to existing actions for TextEdit. Additionally, I decided to expand them a little bit to also allow getting or setting the selection or selected lines in the document, not just the full contents of the document.

The first step was pretty simple. With well-defined, compact actions like the ones I considered, the interface requires little thought and little effort. In my case, it was just a matter of populating the menu entries for a pop-up button and establishing a binding.

To my surprise, I spent the most time and had the most difficulty with the second step. I'd expected this to be easy, but found that there were quite a few details that weren't made especially clear in the description of the keys for the property lists. The analogous actions for TextEdit proved very helpful, here: in several cases, I determined the correct choices by examining the plists from the bundles for the TextEdit actions. I'd expect this step to get easier with practice.

Actually implementing the behavior was especially easy, in this case. As I already had working handlers, I was able to just focus on connecting the action to the handlers. The structure for this is quite pleasant: a record of parameters is injected into the run handler, containing values obtained from the bindings to the NSObjectController. The resulting structure for my actions was then just using if-then statements to interpret the parameters, calling out to different handlers.

Although implementation of the behavior was easy, I did find testing to be quite tedious. It was helpful to save a workflow for testing, as suggested by Neuburg. More importantly than that, I'd strongly recommend that you have working handlers before trying to make them part of your action—test out the handlers in, e.g., Script Editor, and test that your Automator action dispatches to the correct handler.

The actions I created are available, as are the sources.

Setting up SkyBlueCanvas for Local Use on Mac OS X Leopard

2009-02-18T12:00:00.001-08:00

SkyBlueCanvas (SBC) is a content management system (CMS) for web sites. It can be set up to run on a Mac OS X system using Web Sharing, but some steps need to be taken first. The result is a subfolder of your Sites folder, which shows off pretty much the full features of SBC. Unfortunately, you won't get to this point just by naively following the installation instructions for SBC, because it depends on features not enabled by default in Mac OS X.

Note that I'm not making any recommendations as to whether or not you should use SBC, but am instead just sharing what I had to do to get SBC working. This was driven by my own interest in trying out SBC. I have insufficient experience with CMSs in general or SBC in particular to make an informed judgement.

Several of the steps are multi-step themselves. I've just linked to the websites I found that provided a useful description. Total time to set things up should be under half an hour. After the initial set-up, starting a new site can be done by starting with step four, taking under five minutes.

Enable PHP.

Make sure that your Sites folder is reachable.

Enable .htaccess files.

Copy the SkyBlueCanvas folder to your Sites folder, and give it a reasonable name (I used skyblue, and will assume you do the same).

Change the file permissions of the skyblue folder to 755. You can do this in the Terminal with a command like chmod -R 755 ~username/Site/skyblue—replace username with your actual user name, of course. (I didn't actually find this to be necessary, although it is in the SBC installation instructions.)

Edit htaccess.txt in the skyblue folder, adding a second line of RewriteBase /~username/skyblue/. Rename htaccess.txt to .htaccess.

Change the group and owner for the SBC files. In Terminal, change to the skyblue directory, and run sudo chown -R www:www *. If you don't do this, PHP won't be able to make needed changes to the files.

Start Apache. Open up the Sharing panel in the System Preferences, and check Web Sharing.

Point your web browser at http://localhost/~username/skyblue/ and follow the instructions. When asked for the site URL, give http://localhost/~username/skyblue/.

W3R-7: Summary and Assessment

2009-01-17T06:40:00.001-08:00

In the What's Wrong With Reduce series, I've attempted to resolve contrasting views of the usefulness of higher order function called reduce or fold. By examining the origins of reduce, we saw that it is an abstraction over a particular recursive pattern of control flow. Further, reduce can be readily defined in Python.

However, it is easy to find cases where reduce works well for a functional programming language but poorly in Python. In essence, reduce is an abstraction over a common pattern for functional programming languages that is uncommon in imperative programming languages like Python. Instead, a much narrower selection of Python procedures map well onto the reduce control abstraction.

By examining control flow patterns actually used in Python, a more appropriate higher order procedure¹, app, was identified as an alternative to reduce. Several common control flow patterns cannot be captured by app, so a dual gen procedure was also defined.

Between app and gen, we can define many functions to process or create iterators. However, an important question remains: should we use app or gen? I'd have to say that it isn't worthwhile. Both app and gen are abstractions over imperative patterns, depending crucially on side effects. As a consequence, it is of equal importance to understand the order in which, say, app processes the elements of an iterator. Thus, app fails to provide any conceptual benefit over just using a for loop—in contrast, functional programming with reduce, map, and other higher order functions lets you ignore the order in which things are done.

That said, I use and define higher order procedures all the time in Python. I just define them as (generator) functions using imperative constructs like for or while loops. Also, like the itertools module, my higher order procedures typically take iterators as arguments or return them as results. So long as iterators only need to be used once, this looks a lot like functional programming. Really, it is closer to dataflow programming.

Having an effective alternative to reduce is relatively unimportant for this iterator-intensive programming style. Far more useful would be higher order procedures that simplify composing multiple procedures taking and returning iterators. The overall result should be more like constructing a shell pipeline instead of the extensive nesting of procedure calls currently seen.

¹ For clarity, I distinguish here between higher order functions and higher order procedures. Functions should really be mathematical functions, i.e., no side effects, while procedures are more general.

W3R-6: Dualing Reduce

2009-01-08T03:18:00.001-08:00

By examining actual programs, we saw that reduce is the wrong control abstraction in many cases. By starting from programs instead of starting with the abstractions, we saw that app is a more natural control abstraction for Python. The comparatively few cases where reduce is right were handled by introducing an adaptor Accumulator, with app and Accumulator being sufficient to define reduce.

Unfortunately, the approach of defining an adaptor to get app to do what we want leaves out an extremely common way that for loops get used. Consider defining map in terms of the lazy imap¹. It looks something like:

def map_lazy(function, iterable):
    return list(imap(function, iterable))

def imap(function, iterable):
    for x in iterable:
        yield function(x)

All the real work is now done in imap. However, imap cannot be written using app, since there is no way to pass in a function that will yield values we can use².

In imap, both for and yield express control flow, distinct from that given by app. Similarly, consider defining filter using an ifilter:

def ifilter(predicate, iterable):
    for x in iterable:
        if predicate(x):
            yield x

In ifilter, we have another pattern of control flow, regulated with an if statement. Thus, we need an additional higher order procedures if we want to abstract over these patterns. It should be, informally, a dual to app, creating an iterator instead of consuming one.

I don't know of a standard name for the dual procedure—it's a generator (probably confusing to call it that) and has a great deal of similarity to unfolds. At the risk of some confusion, I'll go with the short name of gen. A possibility for writing it is:

def gen(init, update, shouldEmit, emit):
    state = init
    while True:
        if shouldEmit(state):
            yield emit(state)
        update(state)

Note that I've included no explicit treatment of the internal state, so gen can in principle handle any sort of internal state, be it an iterator or otherwise.

I don't think I've seen anything like gen in Python code. Instead, generator functions are written out explicitly on a case-by-case basis. Looking at how we'd implement imap with gen suggests why:

def imap_gen(function, iterable):
    def update(state):
        it = state[2]
        state[0] = it.next()
        state[1] = True
    def emit(state): return function(state[0])
    def shouldEmit(state): return state[1]
    init = [None, False, iter(iterable)]
    return gen(init, update, shouldEmit, emit)

There is quite a lot of effort needed to convert iterable into something that we can manage with gen. A relatively simple class could be given to initialize and update the iterator state, paralleling the approach of app plus an adaptor.

Perhaps there is an alternative to gen that would be more natural for iterators as the internal state, while also cleanly handling non-iterable states. However, I'm not going to pursue the matter further, as gen demonstrates how to encode another pattern of control complementary to that encoded by app (or reduce). Taken together, app and gen can accomplish a great deal, creating an iterator with gen and processing its outputs using app.

¹ Arguably, we don't need map at all when we have imap. This is the case in Python 3, where map returns an iterator.

² This is a limitation of Python's coroutines. More general (and more complicated) routines, such as those in Lua, do allow such an implementation.

W3R-5: Cart Follows Horse

2009-01-06T03:36:00.001-08:00

Higher order functions can be used to implement control abstractions, but that doesn't mean all higher order functions provide useful control abstractions. As we've seen, a useful control abstraction in one language may not work so well in another. In SML, folds are frequently used, while the analogous reduce is uncommon in Python. In retrospect, this doesn't seem so surprising: SML is principally a functional programming language depending on declarative (i.e., recursive) control flow, while Python is a hybrid procedural and object-oriented programming depending on imperative control flow. Where the two languages overlap is where reduce becomes useful, but that overlap is relatively small.

If reduce abstracts over the wrong pattern of control flow, then what are the correct patterns for Python? Let's return to map and see what we can conclude. An imperative definition of map is natural, along the lines of:

def map_imper(function, iterable):
    acc = []
    for x in iterable:
        acc.append(function(x))
    return acc

This version of map utilizes a typical imperative pattern of control, consisting of mutating an object based on values encountered as we iterate though a collection using a for loop. Python has no higher order procedure capturing this imperative pattern; there is such a higher order procedure for Standard ML, where it is called app.

Let's define app for Python. It's easy:

def app(mutate, iterable):
    for x in iterable:
        mutate(x)

Note that app has no useful return value, always returning None, emphasizing that it is always called for its side effects. Using app to define map, we obtain

def map_app(function, iterable):
    acc = []
    def mutate(x): acc.append(function(x))
    app(mutate, iterable)
    return acc

The definition of map_app is no simpler than that of map_imper, but does illustrate an appropriate control abstraction.

Although not needed for map, several generalizations of app could be made. One possibility would be to include optional arguments in order to allow slicing of the iterable:

import itertools
def app(mutate, iterable, start=None, stop=None, step=None):
    if start is not None or stop is not None or step is not None:
        iterable = itertools.islice(iterable, start, stop, step)
    for x in iterable:
        mutate(x)

Many other variations are possible. I'll not suggest which is correct, as that is a question best addressed by empirical analysis of existing code bases.

Now let's reconsider our other motivating example, sum. An imperative implementation is quite similar to that for map:

def sum_imper(numbers):
    acc = 0
    for x in numbers:
        acc += x
    return x

This is the same pattern of control as used in map, mutation within a for loop, but it cannot be expressed using app. In sum, the mutation is that of the variable acc. We can't pass in a variable to app and expect it to be modified, as only the local variable would be changed. That is, app mutates values, while sum mutates a variable.

I can think of two solutions. First, we could introduce another higher order procedure to abstract over what happens in sum. That procedure—function, in this case—is just reduce. Second, we could adapt the value from an immutable number to some mutable object. Which to choose is pretty much a matter of taste. Having two substantially similar control flow mechanisms may be undesirable, but reduce is also a known quantity. The adaptor approach is not as well explored, but may turn out to be more natural. Let's take a look.

When combined with app, the adaptor needs to produce the same effect as reduce. Since app is handling the control flow, the adaptor just needs to handle the accumulation. One possible definition is to use an Accumulator class, defined as:

class Accumulator(object):
    def __init__(self, function, init):
        self.function = function
        self.value = init

    def update(self, x):
        self.value = self.function(self.value, x)

As an example of usage, here is an accumulator for summation:

>>> acc_sum = Accumulator(operator.add, 0)
>>> acc_sum.update(4)
>>> acc_sum.value
4
>>> acc_sum.update(37)
>>> acc_sum.value
41

Putting together an Accumulator instance and app, we can give another definition of sum:

def sum_adapt(numbers):
    acc = Accumulator(operator.add, 0)
    app(acc.update, numbers)
    return acc.value

As a second example, here is yet another definition of a fold:

def fold_app(function, init, iterable):
    acc = Accumulator(function, init)
    app(acc.update, iterable)
    return acc.value

Personally, I find these definitions to be quite natural and appealing; app and Accumulator would fit in well with my programming style.

W3R-4: What's Wrong With Reduce.

2009-01-04T09:11:00.001-08:00

We've used fold—our model of reduce—to define sum, product, and a function min3 that finds the three smallest elements in a list. We have not yet used fold to define map, which was one of the functions we used to motivate fold in the first place. Let's do that now¹.

With linked lists, it is easy:

def reverse(lnkLst):
    def app(acc, x): return Cons(x, acc)
    return fold(app, Nil(), lnkLst)

def map_ll(function, lnkLst):
    def app(acc, x):
        return Cons(function(x), acc)
    return reverse(fold(app, Nil(), lnkLst))

I've introduced a reverse function to preserve the order, as we didn't bother to define a right-to-left foldr, just the left-to-right fold that parallels Pythons reduce. Otherwise, the definition is straightforward. The accumulating function needs to create a new linked list by transforming values from the source list. This is a perfect match for fold.

The same idea can be applied to Python's lists (actually arrays):

def map(function, lst):
    def app(acc, x):
        return acc + [function(x)]
    return fold(app, [], lst)

However, this is quite poor. This implementation of map creates a new list at each step, and thus performs very badly—O(N²) time, where N is the length of the list, instead of the O(N) that should be taken. Even though fold—i.e., reduce—is applicable to any iterable, that does not mean it has captured a useful pattern for every iterable!

So what went wrong? Folds are a good way to break down data structures, which led us to iterators, the standard way to take apart data structures in Python. That worked fine, and we really have no reason to expect it to suddenly fail. However, we have introduced something new: we're also using reduce to build up a new data structure—specifically, a new list. This works fine with the functional link list but fails badly with the imperative arrays.

To see this, let's expand the fold out again, using fold_iter (introduced earlier) to do it:

def map_iter(function, lst):
    acc = []
    for x in iterable:
        acc = acc + [function(x)]
    return acc

The final form is just not how it should be done. In other words, fold is an abstraction over the wrong pattern!

What has happened is that we've gone about things entirely backwards. In a functional programming language like SML, folds are useful. That usefulness was (apparently) the basis for thinking reduce should be useful in Python. However, folds are useful in functional programming languages because they capture a particular pattern of recursion that is common in functional programming. That pattern of recursion is not common in Python, so we have no reason to conclude that reduce will be useful².

¹ For simplicity, I'll only allow a single list to be given as an argument to map, unlike the Python built-in function.

²Nor may we conclude on logical grounds that reduce will not be useful. Experience suggests it is not particularly so, but we don't know if that is something fundamental about reduce or something incidental, such as the documentation.

W3R-3: Object-Oriented Reduce

2009-01-04T04:52:00.001-08:00

We now have worked through deriving reduce in Python. In doing so, we have defined two closely related functions, fold¹ and scan. Since they're so similar, it makes sense to briefly digress and consider bundling them together into a class. I won't make much use of the class in future installments of this series, but it is an interesting exercise to consider alternative definitions that might be a better fit to Python.

Define:

class Fold(object):

    def __init__(self, function, init):
        self.function = function
        self.init = init

    def __call__(self, acc, x):
        return self.function(acc, x)

    def scan(self, iterable):
        function = self.function
        acc = self.init
        yield acc
        for x in iterable:
            acc = function(acc, x)
            yield acc

    def fold(self, iterable):
        for x in self.scan(iterable): pass
        return x

The Fold class is defined to take the function and initial value for the accumulator when the class is instantiated. This gives us some of the benefits we saw for the curried SML functions.

Putting the Fold class to use, we can define the sum and cumsum functions that we saw last time:

sum = Fold(operator.add, 0).fold
cumsum = Fold(operator.add, 0).scan

Perhaps more useful is to define something like<

add = Fold(operator.add, 0)

and then using add.scan and add.fold as needed.

Indeed, any function of two arguments could be given this treatment, so long as a sensible default (or identity) value can be given. Note that the min3 function from last time is not well suited to this sort of treatment, because it has no clear default value. However, the Fold class presents no additional difficulties over separate fold and scan functions.

¹ As a reminder, fold is another name for reduce. We'll continue to reserve reduce as the name of the Python built-in function.

W3R-2: Importing Reduce Into Python

2009-01-02T11:01:00.001-08:00

Last time, we took a look at reduce in Standard ML, where it is known as a fold. Now, we'll translate those folds into Python. Let's keep calling them folds, reserving reduce for describing the existing Python function.

The main challenge in the translation process is that Python doesn't have the right data types—Python's lists are actually arrays, not (linked) lists. As a first step, introduce two classes Cons and Nil to define linked lists:

class Cons(object):

    def __init__(self, head, tail):
        self.head = head
        self.tail = tail

    def __iter__(self):
        lnkLst = self
        while True:
            if lnkLst.isNull():
                break
            yield lnkLst.head
            lnkLst = lnkLst.tail

    def __repr__(self):
        return "Cons(%s, %s)" % (repr(self.head), repr(self.tail))

    def isNull(self):
        return False

class Nil(object):

    def __repr__(self):
        return "Nil()"

    def __iter__(self):
        if False:
            yield

    def isNull(self):
        return True

With these classes, we can construct lists in a Lisp-like fashion:

Cons(0, Cons(1, Cons(2, Cons(3, Cons(4, Nil())))))

For convenience, we'll also introduce a function to convert a Python list into a linked list:

def fromList(lst):
    lnkLst = Nil()
    for x in lst[::-1]:
        lnkLst = Cons(x, lnkLst)
    return lnkLst

Using fromList(range(5)) is a lot easier than nesting Cons repeatedly, as shown above.

With linked lists available, it is easy to define a (left) fold:

def fold_recur(function, init, linkedList):
    if linkedList.isNull():
        result = init
    else:
        acc = function(init, linkedList.head)
        result = fold_recur(function, acc, linkedList.tail)
    return result

I'll note two important differences between the SML folds and fold_recur. First, the arguments to the function taken by fold_recur are reversed from those taken by its SML counterpart, in order to match Python's reduce. Second, and far more importantly, fold_recur is terribly broken because it is a tail recursive function, but only has Python's broken support for recursion to draw on. Trying to use fold_recur will cause stack overflows when the list is too long—that is, containing 1000 elements with the default recursion limit¹.

To rectify this problem, we need to transform fold_recur by hand into an imperative version. Fortunately, fold_recur is constructed in an iterative form, so the transformation is easy:

def fold_imper(function, init, linkedList):
    acc = init
    while not linkedList.isNull():
        acc = function(acc, linkedList.head)
        linkedList = linkedList.tail
    return acc

With the imperative definition, our fold will not cause a stack overflow.

Since we've defined linked lists to be iterable, we can still make our fold a bit more elegant:

def fold_ll_iter(function, init, linkedList):
    acc = init
    for x in linkedList:
        acc = function(acc, x)
    return acc

Although fold_ll_iter was obtained by iterating over a linked list, it will work with any iterable. Rewrite it to make that clearer:

def fold_iter(function, init, iterable):
    acc = init
    for x in iterable:
        acc = function(acc, x)
    return acc

Through this process, the deconstruction of a data structure has been shifted from SML fold functions to Python iterators. This is unsurprising: in Python, it is quite standard to use iterators to deconstruct data structures. It is not completely equivalent, but covers many cases.

The iterator approach suggests another modification of fold, which is normally called scan:

def scan(function, init, iterable):
    acc = init
    yield acc
    for x in iterable:
        acc = function(acc, x)
        yield acc

Our scan function processes the list in the same sequence, yielding values accumulated at each step along the way. Using scan, we define one last version of fold:

def fold(function, init, iterable):
    for x in scan(function, init, iterable): pass
    return x

We can use scan and fold to define many useful functions in a simple manner. For example:

import operator
def sum(numbers): return fold(operator.add, 0, numbers)
def cumsum(numbers): return scan(operator.add, 0, numbers)
def product(numbers): return fold(operator.mul, 1, numbers)
def cumproduct(numbers): return scan(operator.mul, 1, numbers)

I think these functions illustrate rather well the difference between fold and scan. In sum, we use fold to total up the numbers in a list, while in cumsum we use scan to generate the partial sums as we proceed through the list. The product and cumproduct functions are similar.

More complex functions are possible as well. Consider finding the three smallest items in a list. This can be done in linear time by iterating through the list and keeping track of the three smallest as we proceed. But that's exactly what fold is for! The accumulator is a collection of the three smallest items, and we just need a function to update that. Putting that together:

def min3(lst):
    def comp(acc, x):
        a,b,c = acc
        if x < a:
            result = x,a,b
        elif x < b:
            result = a,x,b
        elif x < c:
            result = a,b,x
        else:
            result = acc
        return result
    init = sorted(lst[:3])
    return fold(comp, init, lst[3:])

For simplicity, I've assumed that min3 takes a list, but it could easily be modified to work with any iterable.

I've presented a lot of program code in this post, transforming a direct translation of the SML folds into function that are defined in a more natural way for Python. Next time, I'll digress briefly to present an alternative structure that has some practical advantages over what's been shown here, then return to examining why reduce doesn't see much use in Python.

¹ In practice, this might as well read that the limit is 1000 elements, end of story. The limit can be increased with sys.setrecursionlimit, but "a too-high limit can lead to a crash."

W3R-1: Origins of Reduce

2009-01-01T14:04:00.001-08:00

To begin our investigation of reduce in Python, we need to come to grips with what reduce actually is. I find that the documentation for reduce is rather poor, requiring a fair amount of thinking to determine just what you need to pass in as arguments. Instead, let's examine the origins of reduce to better understand it.

Reduce originates outside of Python. It comes to Python through a contributed patch:

About 12 years ago, Python aquired lambda, reduce(), filter() and map(), courtesy of (I believe) a Lisp hacker who missed them and submitted working patches.

The fate of reduce() in Python 3000
Guido van Rossum, 2005

That's not very helpful, so let's derive reduce by abstracting from some examples. Instead of Lisp, I'll use Standard ML. SML has two key advantages over Lisp: the syntax is more approachable for the uninitiated, and its robust (Hindley-Milner) type system and pattern matching will make things a little clearer. I'll follow the definitions in SML rather than Python, so we'll refer to reduce-like functions as folds, with argument order matching those in the Standard ML Basis Library.

Consider adding up a list of integers. In SML, that list is a linked list, so we'll have a typical recursive solution:

fun sum [] = 0
  | sum (h::t) = h + sum t

I've made use of pattern matching to define sum in terms of two cases, that of an empty list and of a non-empty list. The double colon is a cons operator, joining the head h and tail t of the list.

To define map, a similar recursion scheme is used:

fun map f [] = []
  | map f (h::t) = (f h) :: (map f t)

Map is written as a curried function, so different list processors can be defined by specifying just the function map f, only giving the list later.¹

Just about any function over a list uses the same recursion scheme. Let's extract that recursions scheme into a higher order function called foldr:

fun foldr f acc [] = acc
  | foldr f acc (h::t) = f(h, (foldr f acc t))

Foldr deconstructs a list from right to left, starting with the tail and accumulating the values. As with map, foldr is a curried function.

The type of foldr can make it easier to understand:

('a * 'b -> 'b) -> 'b -> 'a list -> 'b

The first term, in the parentheses, describes the function f; it takes a list element and an accumulator, returning an updated accumulator. The second argument to foldr is an initial value for the accumulator. Third is the list to process. Importantly, the types of the accumulator and the list elements can differ.

We can use foldr to define sum and map. Here's sum:

val sum = foldr Int.+ 0

For map, we'll introduce a helper function and use foldr on that:

fun map f lst = let
  fun maphead(x, acc) = (f x) :: acc
in
  foldr maphead [] lst
end

To get the helper function maphead right, I just looked at the types needed for foldr and wrote the obvious function to satisfy those types.

While foldr allows elegant definition of our two example functions, there is a problem. As written, foldr walks through the entire list before starting to do any work. If the list is long, this can cause a stack overflow. We need a tail-recursive version, leading us naturally to foldl:

fun foldl f acc [] = acc
  | foldl f acc (h::t) = foldl f (f(h, acc)) t

In contrast to foldr, foldl processes the list from left to right, starting with the head. This definition of foldl is tail recursive, so operates in constant space. It can also be used to define a tail recursive foldr by reversing the list before passing it to foldl.

Foldl is substantially similar to reduce in Python. Next time, I'll translate the above definitions to Python, to make the connection transparent.

¹ Having a curried map in SML makes for a big conceptual difference from the map in Python. It's quite reasonable to view map as transforming a function from acting on values to a function acting on lists of values, without map itself actually having anything to do with the list.

What's Wrong With Reduce?

2008-12-31T07:46:00.001-08:00

Functional programming seems to be on the rise—there's even an O'Reilly book on Haskell, now. I'm not quite sure how that increase in popularity happened, but would guess that Paul Graham's advocacy of Lisp had something to do with it, as did Douglas Crockford's work with JavaScript. I also suspect that the higher-order constructs in Python played an important role, allowing imperative programmers to gradually start using a functional style.

However, there is something curious about the functional constructs in Python. One of the higher-order functions in Python—reduce—seems not to find much acceptance. Probably the best example of this is from the BDFL himself, who wrote:

…almost every time I see a reduce() call with a non-trivial function argument, I need to grab pen and paper to diagram what's actually being fed into that function before I understand what the reduce() is supposed to do. So in my mind, the applicability of reduce() is pretty much limited to associative operators, and in all other cases it's better to write out the accumulation loop explicitly.

The fate of reduce() in Python 3000

Guido van Rossum, 2005

I don't think the above is out of line with most Python users think, but it clashes severely with conventional thinking of users of functional programming languages.

For example, in an influential paper, John Hughes wrote

…a little modularisation can go a long way. By modularising a simple function (sum) as a combination of a “higher order function” and some simple arguments, we have arrived at a part (reduce) that can be used to write down many other functions on lists with no more programming eﬀort.

and

…functional languages allow functions which are indivisible in conventional programming languages to be expressed as a combination of parts—a general higher order function and some particular specialising functions. Once deﬁned, such higher order functions allow many operations to be programmed very easily. Whenever a new datatype is deﬁned higher order functions should be written for processing it. This makes manipulating the datatype easy, and also localises knowledge about the details of its representation. The best analogy with conventional programming is with extensible languages—it is as though the programming language can be extended with new control structures whenever desired.

Why Functional Programming Matters

John Hughes, 1984

My own experience with ML is quite in line with what Hughes wrote: reduce, or foldl as it is also known, is used all the time, and seems very natural.

Why the difference? I'm not aware of any case where GvR has demonstrated much knowledge of functional programming—maybe he was just speaking outside his area of expertise, and got it wrong? The PLT Scheme people may have thought so, satirizing him shortly after in a (quite funny!) April Fool's Day announcement.

However, I think that explanation is overly facile. Personally, while I use folds in functional languages, I rarely do in Python. I think there are real reasons for that, and will examine them in future blog posts.

Power Laws and Social Problems

2008-12-10T03:01:00.001-08:00

Million-Dollar Murray is a fascinating article by Malcolm Gladwell about how "power laws" (really, heavy tailed distributions) can have surprising policy implications for dealing with social problems.

The article focuses most strongly on homelessness. The core of how power laws clash with our assumptions is summarized as:

That is what is so perplexing about power-law homeless policy. From an economic perspective the approach makes perfect sense. But from a moral perspective it doesn't seem fair. Thousands of people in the Denver area no doubt live day to day, work two or three jobs, and are eminently deserving of a helping hand—and no one offers them the key to a new apartment. […] Social benefits are supposed to have some kind of moral justification. We give them to widows and disabled veterans and poor mothers with small children. Giving the homeless guy passed out on the sidewalk an apartment has a different rationale. It's simply about efficiency.

Another issue considered is police brutality. After the Rodney King beating in Los Angeles, a special commission headed by Warren Christopher investigated the LAPD:

But what was the commission's most memorable observation? It was the story of an officer with a known history of doing things like beating up handcuffed suspects who nonetheless received a performance review from his superior stating that he "usually conducts himself in a manner that inspires respect for the law and instills public confidence." This is what you say about an officer when you haven't actually read his file, and the implication of the Christopher Commission's report was that the L.A.P.D. might help solve its problem simply by getting its police captains to read the files of their officers. The L.A.P.D.'s problem was a matter not of policy but of compliance. The department needed to adhere to the rules it already had in place, and that's not what a public hungry for institutional transformation wants to hear. Solving problems that have power-law distributions doesn't just violate our moral intuitions; it violates our political intuitions as well.

The third issue explored is pollution by automobile exhaust. This seems quite prosaic in comparison to the first two, but may be the best illustration of the way that we can incorrectly address a problem if we have an erroneous view of the distribution of the problem. For cars, we have:

Most cars, especially new ones, are extraordinarily clean. A 2004 Subaru in good working order has an exhaust stream that's just .06 per cent carbon monoxide, which is negligible. But on almost any highway, for whatever reason—age, ill repair, deliberate tampering by the owner—a small number of cars can have carbon-monoxide levels in excess of ten per cent, which is almost two hundred times higher. In Denver, five per cent of the vehicles on the road produce fifty-five per cent of the automobile pollution.

Given this heavy tailed distribution, alternatives to the customary annual inspection become more relevant:

[Donald Stedman, a chemist and automobile-emissions specialist at the University of Denver] proposes mobile testing instead. Twenty years ago, he invented a device the size of a suitcase that uses infrared light to instantly measure and then analyze the emissions of cars as they drive by on the highway. […] He says that cities should put half a dozen or so of his devices in vans, park them on freeway off-ramps around the city, and have a police car poised to pull over anyone who fails the test. A half-dozen vans could test thirty thousand cars a day. For the same twenty-five million dollars that Denver's motorists now spend on on-site testing, Stedman estimates, the city could identify and fix twenty-five thousand truly dirty vehicles every year, and within a few years cut automobile emissions in the Denver metropolitan area by somewhere between thirty-five and forty per cent. The city could stop managing its smog problem and start ending it.

Not mentioned by Gladwell is that this mobile testing would also have a moral benefit: costs could be shifted to polluters, instead of spreading the cost to everyone.

In summary, a fascinating article that connects a current scientific issue with current societal issues. I'm sure it only scratches the surface. No doubt, many other issues depend crucially on the distribution of underlying factors.

Formatted Program Text for Blogger

2008-12-08T13:20:00.001-08:00

This blog features a fair amount of code. So far, I've been doing the formatting from SubEthaEdit. Using SEE has worked OK, but was kind of a fiddly procedure. I'd open the desired program, copy as XHTML, paste that into a new document, remove all the lines breaks, copy the modified text, and paste it into MarsEdit. No step is difficult, but automating it is a pain, requiring GUI scripting in AppleScript. Further, it would require running everything through SEE, whether that would make sense or not.

Thus, before scripting the above approach, I looked for other options. Two seem promising: Pygments and highlight. Both provide shell filters for converting program text into HTML. For now, I've settled on highlight, since Pygments lacks support for highlighting Standard ML.

Several command line options are appropriate. For example, I can format an SML file with

cat real-ord.sml | highlight --inline-css -f --enclose-pre -S sml -s seashell | pbcopy

Pasting that into MarsEdit nicely formats the contents of real-ord.sml:

structure RealOrdKey : ORD_KEY where type ord_key = Real.real = struct
  open Real
  type ord_key = real
end

I'm sure I'll need to refine the approach a bit as I go, but this already seems easier than the approach I'd been using.

Another Skeptical Take on Python 3

2008-12-08T09:00:00.001-08:00

Jens Alfke wonders "What’s The Point?" of Python 3.

Via Daring Fireball.

Thoughts on Python 3.0

2008-12-05T09:00:00.001-08:00

Python 3.0 has been released. I first used Python at either the end of 1999 or the beginning of 2000. It's with some surprise that I find myself rather ambivalent about what is a major landmark for a tool that I've used on a near-daily basis for almost nine years.

There are a variety of reasons for my ambivalence. Largely, my own programming interests have diverged from the main features of Python. Thus, the significant changes that are present in Python 3 just aren't addressing the shortcomings of Python that affect me. In particular,

My interest in functional programming has continually increased since I first learned about it in 2002. Python is pretty limited in its support for functional programming, especially in its inability to do tail call optimization and its lack of suitable data structures.
Learning about functional programming led me to Standard ML and its kin. Reading Harper's Standard ML book profoundly affected how I think about programming, perhaps even more than did reading SICP. Much to my surprise, I found that static typing didn't have to be the nightmare I remembered from C programming. Indeed, typeful programming turns out to be a natural match for how I like to work, with types encoding—and enforcing—a great deal of the assumptions that go into function definition. Python, with its untyped variables, doesn't lend much support to this style.
Concurrency appears increasingly relevant. I'd like to learn more about concurrent programming, and apply it in practice. Specifically, message passing concurrency shows great promise, interacting well with a functional programming style and providing ready control over when nondeterminism appears (see, e.g., a recent post on Lambda the Ultimate). Python isn't structured for the approach I want to explore.

Basically, there are things that I want to do that Python just gets in the way of, and the future of Python seems quite unlikely to include the changes I need.

In light of the above, the landmark Python 3 release comes across as a good time to re-assess how I use Python. As a whole, I'm left thinking that I'd be better off directing more of my time to Scala, OCaml, or Haskell. The changes in Python 3 hardly seem worth even the relatively minor effort needed to upgrade at this point, so I think I'll just stick with the installation I have for now.

Passive-Aggressive Python

2008-11-18T12:44:00.001-08:00

Consider sum. It's great for adding up a list of numbers:

Python 2.5.1 (r251:54863, Feb  4 2008, 21:48:13) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> x = range(10); print x
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> sum(x)
45

Strings, on the other hand, it doesn't care for:

>>> y = list("abcdefghi"); print y
['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i']
>>> sum(y, "")
Traceback (most recent call last):
  File "", line 1, in 
TypeError: sum() can't sum strings [use ''.join(seq) instead]

This is about performance. Repeatedly adding up strings behaves quite a bit differently than adding up numbers. Each addition causes a new string to be created, turning the whole thing into O(n²), where n is the number of strings being concatenated, instead of the O(n) you would get by using join or when summing up numbers.

Think that seems like a fair tradeoff? Think again:

>>> u = map(lambda a: [a], x); print u
[[0], [1], [2], [3], [4], [5], [6], [7], [8], [9]]
>>> sum(u, [])
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>>

and again:

>>> v = map(lambda a: (a,), x); print v
[(0,), (1,), (2,), (3,), (4,), (5,), (6,), (7,), (8,), (9,)]
>>> sum(v, ())
(0, 1, 2, 3, 4, 5, 6, 7, 8, 9)
>>>

Both of those have the same performance issues with sum that strings have.

So much for polymorphism. The interface is muddled, being neither specialized to numbers, nor polymorphic to objects that support addition. It's all quite silly, really, because there's no reason for sum to behave like this. The worst-case time complexity is intended to be O(n), but easily becomes O(n²) as shown above—and actually is entirely unpredictable because objects can make arbitrary calculations. On the other hand, the TypeError raised for a string is an arbitrary restriction, since it could instead just internally make use of ''.join.

Apparently, sum is the way it is because that's just not how one should concatenate a string. Anything else that supports addition, go ahead, but for strings there is one—and only one—right way to do it. By fiat.

Phrase from nearest book

2008-11-13T00:23:00.001-08:00

Odd idea that I've seen first on Grig Gheorghiu's blog. Here are the rules:

Grab the nearest book.
Open it to page 56.
Find the fifth sentence.
Post the text of the sentence in your journal along with these instructions.
Dont dig for your favorite book, the cool book, or the intellectual one: pick the CLOSEST.

For me, the nearest book was Practical Statistics by Russell Langley, giving this:

The average duration of numbness in these tests was 50% longer with Prilocaine than with Lignocaine, but the lack of any information about the degree of dispersion around these arithmetic means makes it impossible to assess the significance of the observed difference.

It may not be terribly poetic, but it is a quite profound idea in statistics.

Certainly More Interesting than Anything I've Ever Done with LaTeX

2008-10-15T10:03:00.001-07:00

That is just insane. But in a good way!

My Results for the Sarah Palin Baby Name Generator

2008-09-16T13:42:00.001-07:00

The Sarah Palin Baby Name Generator has been mentioned by many sites lately, so a description is probably not needed. My name would be Thump Hummer Palin.

An Anniversary, of Sorts

2008-08-01T03:53:00.001-07:00

It has been one year since the O'Reilly MacDevCenter site has published an article on Cocoa. The articles since then are a mix of various Mac OS X tips and tricks, with a heavy emphasis on Aperture—so no actual Mac development articles.

It's been quite a while since O'Reilly has published a book on Mac development, too. There is the Missing Manual series, but it is really something distinct. Has O'Reilly given up on Mac developers?

AppleScript Syntax: Dealing with the Depressing Reality

2008-07-27T12:36:00.001-07:00

Given the preceding two posts, I hope it is clear that I think syntax is of too much concern in AppleScript discussions. The preoccupation with syntax can hide real problems that might actually be sensibly discussed. It seems appropriate to mention some of the aspects of AppleScript that I think could be usefully discussed. I'll limit myself here to points where I can provide some practical tips as starting points.

To begin, name resolution is pretty tricky. Rather than some sensible lexical scoping, there are complex rules. John Gruber provides an excellent description of how names are resolved, in the context of explaining a subtle bug. There is a fact that I've found useful for avoiding name conflicts: you do not have use tell blocks.

Frequently, you just need to tell applications to do a few things. However, example code often has the entire program in a single tell block. That leads to all the tricky rules for determining when a term is looked up in an application dictionary, or a Scripting Addition, or as a handler in the script. Many of these issues can be eliminated by doing something like tell application "Finder" to … instead of using the block format. In essence, this is the same idea as avoiding the use of an implicit this or self object, as seen in some object oriented languages.

Said approach also is particularly enlightening on how much of your program is actually spent dealing with interprocess communication, and how much is program logic. Frequently, not much is actually spent on dealing with IPC, but it leads to a lot of complications. Simple solution: encapsulate IPC in handlers.

Text handling is important in many scripts. AppleScript is pretty weak in its text handling. Mac OS X comes with a powerful text processing system: the Unix shell. Use do shell script for text manipulation.

AppleScript is lacking in the data structures it provides. The one I most frequently miss is an associative array, but it's not the only one. Many other languages provide a richer library of data structures, better support for defining your own, or both. You could often dispense entirely with AppleScript, except for communicating with applications, which might just be a few lines. You can communicate with applications using the osascript shell command. This lets you use just about any language you like, and still be able to gain the main benefit of AppleScript.

There must be many more.

AppleScript Syntax: Some Depressing Examples

2008-07-27T11:03:00.001-07:00

Let's expand on the previous post by looking at some examples of how discussion of syntax poisons discussion of AppleScript. Just to be clear, the examples are taken from blogs I like and read regularly; I'm sure there are many other examples, but I haven't gone out of my way to find these.

First, let's return to Daring Fireball. John Gruber recently wrote that

AppleScript, as a programming language, is a noble but failed experiment.

To support this, he links to an earlier article, The English-Likeness Monster, in part of which he makes the far more modest claim that AppleScript's English-like syntax is a failed experiment.

Well, which is it? Languages are more than their syntax. Reading the article doesn't clarify much. Gruber details how a bug he experienced was caused by a subtle name conflict in scripting dictionary terminology. He gives a cogent description of the semantics of name resolution in AppleScript, showing how the name resolution semantics leads to the bug he experienced. However, buried within is a lengthy rant on AppleScript's syntax. It looks like, and is identified as, a digression (well, it is called an "interpolation"—Gruber is a David Foster Wallace fan, as I recall), but gives the post its title. Just what is the intent?

Additionally, there is mention of Python and JavaScript as having clearer, if more abstract, syntax than AppleScript, which helps to prevent such problems. However, if you had AppleScript's syntax but Python's name resolution, you literally could not have the same error. There is a difference at a far deeper level than the syntax. To what extent are these articles supposed to be about syntax, semantics, surrounding tools, libraries (i.e., scripting additions),…?

As a second example, let's take a look at an interesting article from Daniel Jalkut's Red Sweater Blog, called Apple's Script. Jalkut makes what is essentially an economic argument that Apple should make JavaScript the default scripting language for the Mac, keeping AppleScript as an alternative point of access to the Open Scripting Architecture. His key point is that Apple is devoting significant resources to JavaScript, and will continue to do so for strategic reasons. Further, due to wide-spread experience with JavaScript, it is in practice easier for users, despite the supposed ease of use of AppleScript (aside: I dispute that AppleScript is easy to use).

Nothing in the article depends in any way on the syntax of AppleScript, but look through the responses! Several people bring up AppleScript syntax, both as a positive and as a negative. Once the issue of syntax appears, the discussion pretty much stays there. It's a shame, because I think that Jalkut's point was an interesting one, and really does warrant some thought.

AppleScript Syntax: the Depressing Corollary

2008-07-27T10:08:00.001-07:00

John Gruber at Daring Fireball links to William Cook's paper on the history of AppleScript. I've been meaning to write about the paper for a few weeks, based on a few recent blog posts relating to AppleScript. The history contains several facts that, in my opinion, are vital for understanding what AppleScript is today, and how we should approach it. In particular, there are some important points to be learned about AppleScript's syntax.

I first read the paper a couple of years ago, after learning about it from Lambda the Ultimate. William Cook is one of the original developers of AppleScript. He describes the development of the language for the third History of Programming Languages conference. The history of a programming language sound likely to be rather dull, but Cook's paper is far from dull. Cook sheds light on many aspects of AppleScript, making clear that the language is a mix of successes and failures. What also becomes clear is that the designers were willing to try genuinely new ideas, not all of which worked out due to practical considerations. As Cook writes,

AppleScript was developed by a small group with a short schedule, a tight budget and a big job. There was neither time nor money to fully research design choices.

It should not be surprising that some of those design choices were suboptimal or even failures.

AppleScript's natural language syntax was one of those failures. Cook writes

The experiment in designing a language that resembled natural languages (English and Japanese) was not successful. It was assume[d] that scripts should be presented in “natural language” so that average people could read and write them.… In the end the syntactic variations and ﬂexibility did more to confuse programmers than to help them out. The main problem is that AppleScript only appears to be a natural language. In fact[, it] is an artiﬁcial language, like any other programming language.… It is easy to read AppleScript, but quite hard to write it.

(I've corrected a few typos that were in the copy of the paper I have, which was an early draft.) Besides making AppleScript accessible to average people, there were additional goals for the natural language syntax. None of them were successful.

In his conclusion, Cook writes

Many of the current problems in AppleScript can
be traced to the use of syntax based on natural language…

Sadly, many critics of AppleScript would have that be the whole story. It is not, and acting as if it were hinders understanding of real problems with AppleScript, some of which could be addressed without changing the syntax at all. In fact, I will assert that the biggest problem with AppleScript's syntax is that it prevents meaningful discussion of AppleScript!

I propose a variant of Godwin's law for AppleScript:

As an online discussion of AppleScript grows longer, the probability of the discussion devolving into a debate on the merits of AppleScript's syntax approaches one. At this point, nothing meaningful will be said, and the discussion is effectively over.

Too frequently, discussion of AppleScript actually begins on the topic of syntactic merits. This leads to the depressing corollary:

Most discussions of AppleScript consist only of a debate on the syntactic merits of the language. There is nothing to be learned from these discussions.

Or, more simply:

Most discussions of AppleScript contain nothing of value.

While harsh, I do feel these are an accurate description of most online (and, for that matter, offline) discussions of AppleScript that I've seen.

A Small Tip on Catch All Address and Spam Blocking in Google Apps

2008-06-25T08:56:00.001-07:00

I've recently signed up for Google Apps, based largely on an article on Lifehacker. One interesting point was the idea of using a catch-all email address to block spam, as described in a comment or with somewhat more detail in a different article. Both explanations omit a relatively minor point that can lead to some frustration.

To summarize the method, you just set up filters in a catch-all address to identify email that matches a simple pattern, and forward it to your own account. The result is that you can give out email address "aliases" for, e.g., web registration forms, without inviting spam to your real address. If you start getting spam to one of the aliases, you can block it directly with a filter.

I set this up, and sent a test message to try it out. The message showed up in the catch-all account, but didn't get forwarded to my own account. The filter I used to set up the forward was identifying the messages correctly, but didn't send it on for some reason.

After a fair amount of web searching, I found the answer. Email for Google Apps is Gmail. Gmail does some modestly smart things to present a convenient interface; perhaps the best known is the way Gmail groups messages into "conversations." Something else it does is to hide messages that you've forwarded to yourself. This seems quite sensible in the context of conversations.

It's probably clear what was happening, at this point. I sent some test messages from my own account, which was then forwarded back to me from the catch-all account. Gmail sees a messages forwarded to myself, and doesn't show it. I'm left scratching my head in puzzlement.

After finding out about the hiding of self-forwards, I try sending a test message from a different account. It forwards without any trouble. Thus, for testing, you should always use a different account to send email than the account you want to receive it in.

Plaxo Synchronization Oddity

2008-06-07T09:23:00.001-07:00

I've used Plaxo for a while to synchronize my address book across a couple of Macs and a web mail account or two. This was the original intent for Plaxo, and it has served me reasonably well as a replacement for the .Mac synchronization (which didn't serve me well, and was overpriced to boot). Today, I discovered a rather strange bit of behavior in the syncronization.

Since I started using it, Plaxo has added a feature they call Pulse. Pulse seems like an interesting variant on the social networking websites, focusing on allowing you to connect up feeds from the sites you actually use and centralize the social networking aspects (e.g., from Blogger, Flickr, and del.icio.us). I set up Pulse today.

Without Pulse, I never had much reason to take a look at the online version of my address book. With Pulse, I could see my own address plainly while setting up my profile. It was wrong.

That was quite strange. The addresses at Plaxo came from synchronizing with the Address Book data on my Mac. Those were right! However, there was a chunk of my previous address showing up on Plaxo, mixed in with my current address.

Here's what happened. I now live in Vienna. The address format for my Address Book card is thus set to Austrian. Before coming to Vienna, I lived in Portugal. The Portuguese format has a field for territorial subdivision, while the Austrian one doesn't use that. The extra stuff showing up in my current address is just the old territorial subdivision. Switching the display format for my Address Book card back to Portuguese shows the territorial subdivision field again. The old data is still there, just hidden with the Austrian formatting. It seems that Plaxo doesn't handle this situation correctly, treating all the data as still relevant (to be fair, I don't know if it is even possible for Plaxo to handle it right).

Admittedly, the described case is a bit unusual. Regardless, it may affect others. The solution, fortunately, is simple: switch the Address Book card format back to the earlier country setting, delete the hidden, outdated fields, switch back to the correct country setting, and sync with Plaxo.

Update: Or maybe Pulse doesn't let you connect a feed from Blogger. Mine doesn't seem to be working, anyway.

Tradeoffs for Type Inference

2008-05-01T05:32:00.001-07:00

Here's a nice essay by Chris Smith: What to Know Before Debating Type Systems. Smith gives a nice introduction to some of the ideas and terminology for programming language type systems, aimed at the perennial static vs. dynamic debate. I quite like that he dismisses the strong vs weak typing distinction as meaningless. Some terminology that I'd have like to have seen is that of untyped languages and type safe languages, since so-called dynamic typing is just that: untyped, but type safe through dynamic checking. I suppose that would have caused a lot of, e.g., Python users to stop reading, which would be a shame.

For the most part, I think Smith is right on with his presentation, although there are a few points where I'd quibble about details. However, I think that type inference warrants some further consideration. As Smith states, "Type inference is generally a big win." I fully agree with that, but there are some details worth making explicit.

First, let's distinguish type inference from what we might call type propagation or local type inference. Type inference is figuring out the types of the expressions from the expressions themselves. Type inference is seen mostly in functional programming languages such as Standard ML, Objective Caml, and Haskell. For the most part, you don't need to write down the type signature for a function, you just define it and the compiler can work out the types. Despite this, it is common to give the type signatures for functions anyway, since the types provide a great deal of information. The type information is useful for understanding how to call and compose functions, which is obviously important for functional programming. Further, declaring the type for a function can be a useful design aid, allowing a correctly typed function stub to be provided that will just raise an exception when called. In Standard ML, such a stub is defined like fun stub x = raise Fail "Not implemented". The inferred type of stub is 'a -> 'b, which is too general to be useful in any real function. By manually giving a type to stub, you can use it to define other functions and the compiler will be able to do more useful checks.

Type propagation is using known types to figure out the types of as many expressions as possible. The latter might take the form of declaring the types for functions and allowing the compiler to work out the types for all the expressions within the function bodies. If you would normally write out the types for functions anyway, you aren't really out much. You might have to write a few more type signatures, since you'd probably skip the signatures for little helper functions, but those types should be easy, anyway. There can also be gains from this approach, since you can use a more expressive type system for which type inference might not be decidable. Such an approach is taken in the local type inference for Scala. There's no guarantee that types can be inferred, but they usually can be so you don't need to write type declarations for every variable. On the other hand, by softening the guarantees for inference, types can be used to do more. It can even fit in with untyped languages for optimization purposes; my understanding is that this was a big part of how Dylan compilers were able to produce fast executables.

As a whole, having type inference provides not just benefits, but also places restrictions on the type system. Improving the type system may well make it worth giving up a general type inference algorithm in favor of weaker type propagation. The marginal cost of annotating some extra functions with types is just not a high price to pay. Turning it around, one really has the worst of both worlds with languages like C, C++, and Java: you have to declare the type of every variable, but it really doesn't provide you with any benefit.

(Via Lambda the Ultimate.)

Code indentation in SubEthaEdit

2008-04-24T13:41:00.001-07:00

There's a recent message on the Yahoo! group for SubEthaEdit asking about automatic formatting of code in SubEthaEdit. By building on SubEthaEditTools, it only takes about twenty minutes to connect an appropriate shell script to any given mode, calling out to, e.g., indent for the C mode. But we can do better!

It takes only a little more effort to install a general system for code reformatting into SubEthaEdit. First, we create a suitable AppleScript that calls out to a formatter:

property prettyPrinter: "${PRETTYPRINTER}"

if not documentIsAvailable() then
    return
end

if (modeSetting for "PRETTYPRINTER") is missing value then
    display alert "Unable to re-indent." message "You need to define PRETTYPRINTER for the mode."
    return
end

if selectionIsEmpty() then
    setDocumentText to (beautifulCode from documentText())
else
    extendSelection with extendingFront and extendingEnd
    setSelectionText to (beautifulCode from selectionText())
end if

on beautifulCode from sourceText
    shellTransform of sourceText for modeEnvironment() through prettyPrinter without alteringLineEndings
end beautifulCode

on seescriptsettings()
    {displayName:"Re-Indent Lines", shortDisplayName:"Re-Indent", keyboardShortcut:"^@i", toolbarIcon:"ToolbarIconRun", inDefaultToolbar:"no", toolbarTooltip:"Automatically re-indents lines", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

The formatter is stored as a PRETTYPRINTER environment variable. The include at the very end is an m4 command, not AppleScript; if you don't know what it's for, you can either just cut and paste the contents of SubEthaEditTools into the script at that point, or read about it here.

The logic of the AppleScript is pretty straightforward. I first check to make sure a document is open, then check to see if a formatter is defined for the mode through a PRETTYPRINTER environment variable. I didn't do these sorts of checks in other scripts for SubEthaEdit, because those were always for just one mode, so I could assume that there was an open document and provide a sensible default for the mode. Here, we're going to put the script into the general scripts folder ~/Library/Application Support/SubEthaEdit/Scripts, so we can't make those assumptions anymore.

At this point, I've made all the safety checks needed, so I can proceed to do the reformatting. If no text is selected, I pass all the text to the beautifulCode handler, otherwise I extend the selection to full lines and pass the new contents to beautifulCode. All that happens in beautifulCode is that the text is handed off to the shell for formatting, with a suitable environment obtained from the mode. The results of beautifulCode are then substituted for the original text.

Finally, seescriptsettings are defined. I set a keyboard shortcut of Command-Control-i; it was available, and allows a mnemonic of i for indent. I defined a toolbar item for it, with the run icon, which is pretty arbitrary; I didn't see anything better from the standard list. It can also be invoked within a document using the contextual menu. Remember, this goes into the general scripts, so appears under the scripts menu, not the Mode menu. I saved mine as PrettyPrint.scpt.

One thing remains: how do we define the PRETTYPRINTER for a mode? It's remarkably easy. The script I used for setting the LaTeX mode environment doesn't actually depend on the LaTeX mode in any way, so it can be used directly by just putting it in the general scripts folder instead of within a mode. Actually, it is probably a good idea to also check that a document is open so that a mode can be identified. With SubEthaEditTools, there's almost nothing to it:

if not documentIsAvailable() then
return
end

openEnvironmentSettings()

on seescriptsettings()
return {displayName:"Customize Mode...", shortDisplayName:"Environment", inContextMenu:"no"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

This script, too, goes into the general scripts folder for SubEthaEdit; I named mine OpenEnvironment.scpt. Although this second script is quite simple, it has a very profound effect, adding a preferences system for all the SubEthaEdit modes. I wonder what else it could be used for?

And that's it. Install the two scripts, define some appropriate PRETTYPRINTER values (being sure to quote the shell command appropriately!), and reformat away. For your convenience, you can download compiled scripts for the formatter and the mode environment settings.

Very Slick Video

2008-04-19T01:14:00.001-07:00

I'm not a big fan of Rush, but this video is quite fun, bordering on stunning.

Watch it.

(Via Pharyngula.)

SEEing LaTeX 29: Listening to LaTeX

2008-04-12T11:49:00.001-07:00

The SubEthaEdit mode I've developed in this series of posts is generally rather quiet. It doesn't announce successful production of a PDF from a LaTeX file, it just creates it and opens it. It also doesn't tell you when an error occurs.

You can have the errors reported by piping the output from the call to pdflatex into some command line tool. Exactly what tool to use is less clear. The natural choice for SubEthaEdit would be the see command line too, but it's not very convenient, and has some real problems. Beyond that, calling out to see repeatedly will produce a new window each time, which is just annoying.

The Python mode has a nicer behavior. It has a Check Syntax command that always writes to a window named Python Check Syntax, opening it if need be or overwriting the contents otherwise. The Lua mode has an analogous Check Syntax command with similar behavior.

Let's make a shell script that does something similar. First, I encapsulate the relevant bits of AppleScript from the Python and Lua modes into some simple handlers. Second, I embed those into a shell script using the approach recently discussed here. The resulting script I call seeless, as it is usable as something like the less pager. I'll defer the text of the script until the end, first describing the usage.

Basic usage is just to pipe in some text:
echo Hello World! | seeless
This writes the text Hello World into a SubEthaEdit window titled seeless message, opening a new window if necessary or replacing the text in an existing window. Multiple windows with the same title are a bit problematic, with no guarantee that the window you want will be the one written to. Don't do that.

The title of the window can be specified:
echo Hello World! | seeless -t"A message for you, direct from the shell"
Appropriate choice of title allows a SubEthaEdit mode to establish its own reporting window.

The window is normally brought to the front when it is written to. Call seeless as
echo Hello World! | seeless -b
to leave the window in the background. This allows, e.g., a reporting window to be kept out of the way as a tab and only checked when something seems to be wrong.

There are two different modes, insert and append, for writing to the window. For insert mode, the window is cleared before writing the text from stdin, while append mode just appends to any existing text. To set append more, use:
echo Hello World! | seeless -a
Insert mode is the default, but a flag exists for it, too:
echo Hello World! | seeless -i
Multiple flags for the insert and append mode can be given, with the final one determining the behavior.

In append mode, the text from a second call to seeless follows immediately after the text from the first. To give some visual space, specify a separator:
echo Hello World! | seeless -s"----------"
When a separator is given with the -s flag, append mode is automatically set. It is also possible to use a separator in insert mode, giving a form of header. For example,
echo Hello World! | seeless -s"$(date)" -i
shows the time when seeless was called. Another possibility would be to have a cluster of related programs writing to the same window, and using the separator to specify the program that wrote the latest text.

Let's take a look at how I put those options into use with the LaTeX mode. I set SEE_BIBTEX to:

'bibtex "${FILE%.tex}" | "$HOME/Library/bin/seeless" -t"LaTeX Messages" -s"bibtex ran at $(date)\n" -b -i &> /dev/null &'

SEE_LATEX_CLEANUP to:

'latexmk -C "$FILE" | "$HOME/Library/bin/seeless" -t"LaTeX Messages" -s"latexmk -C ran at $(date)\n" -b -i &> /dev/null &'

and SEE_LATEX_COMPILER to:

'latexmk -pdf -quiet "$FILE" | "$HOME/Library/bin/seeless" -t"LaTeX Messages" -s"latexmk ran at $(date)\n" -b -i &> /dev/null &'

Note that I have redirected the output of each call to seeless to /dev/null and made the calls asynchronous with &—SubEthaEdit hangs without doing this, requiring a force quit.

With the above settings, the LaTeX mode will cause SubEthaEdit to open up a report window titled LaTeX Messages whenever a document is typeset, bibtex is run, or the auxiliary files are cleaned up. I can put it out of the way, either as a tab or background document; because I've used the -b flag for each call, the report window will stay out of the way until I want it brought to the front. I've used a separator to show which feature was most recently used and at what time it was called. I've set insert mode with an -i flag, so I only see the latest call; by eliminating this flag, I'd have a chronological record of all the calls made (in the current editing session, anyway).

I haven't been using this very long, so there may be some bugs. However, it seems quite solid, and is definitely useful already. Download it here.

As promised above, I'll give the text of the script here, too. I've formatted the script as a shell script, to better show how the shell variables are used to adapt the behavior of the embedded AppleScript.

#!/bin/sh
 
 # Writes stdin to a SubEthaEdit document, modifying the contents if the
 # document already exists. Document is selected by title, with a default
 # title of 'seeless'. Title can be specified with a '-t' flag. If there 
 # are multiple documents with the given title, one is chosen arbitrarily.
 #
 # Writing to the document occurs in two modes, insert and append. With
 # insert mode, the document is cleared before any text is written to 
 # it. In append mode, any existing text is maintained, with the new
 # text appended at the end. By giving an '-a' flag, append mode is set.
 # Giving an '-i' flag sets insert mode; insert mode is the default.
 #
 # A separator can be specified. The separator is written to the 
 # document before the text from stdin. Specifying a separator also sets 
 # append mode (but this can be turned off again if desired with an '-i' 
 # flag). The separator is specified with an '-s' flag; the argument 
 # following the flag is used as the separator.
 # 
 # Giving an '-h' flag shows a usage summary. Any other flags are ignored.
 #
 
 #$Id: seeless.sh,v 1.9 2008/04/12 20:33:56 mjb Exp $
 
 # Copyright (c) 2008, Michael J. Barber
 # 
 # Permission is hereby granted, free of charge, to any person obtaining
 # a copy of this software and associated documentation files (the
 # "Software"), to deal in the Software without restriction, including
 # without limitation the rights to use, copy, modify, merge, publish,
 # distribute, sublicense, and/or sell copies of the Software, and to
 # permit persons to whom the Software is furnished to do so, subject
 # to the following conditions:
 # 
 # The above copyright notice and this permission notice shall be
 # included in all copies or substantial portions of the Software.
 # 
 # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
 # EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
 # MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
 # IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR
 # ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
 # CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
 # WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
 
 
 PROGRAM=$(basename $0)
 
 usage()
 {
     echo "Usage: $PROGRAM [-ahi] [-t title] [-s separator]"
 }
 
 title="$PROGRAM message"
 shouldClear=true
 shouldSeparate=false
 shouldBringToFront=true
 separator="----------"
 while getopts :t:ias:bh opt
 do
     case $opt in
     t)      title="$OPTARG"
             ;;
     i)      shouldClear=true
             ;;
     a)      shouldClear=false
             ;;
     s)      separator="$OPTARG"
             shouldSeparate=true
             shouldClear=false
             ;;
     b)      shouldBringToFront=false
             ;;
     h)      usage
             exit 0
             ;;
     '?')    echo "$PROGRAM: invalid option -$OPTARG" >&2
             usage >&2
             exit 1
             ;;
     esac
 done
 
 shift $((OPTIND - 1))
 
 /usr/bin/osascript > /dev/null <<ASCPT
     set newContents to "$(cat | sed -e 's/\\/\\\\/g' -e 's/\"/\\\"/g')"
     set seeDoc to (ensureSEEDocumentExists for "$title")
     if $shouldClear then
         replaceContents for the seeDoc by ""
     end if
     if $shouldSeparate then
         extendContents for the seeDoc by the "$separator"
     end if
     extendContents for the seeDoc by the newContents
     if $shouldBringToFront then
         tell application "SubEthaEdit" to show the seeDoc
     end if
     
     to ensureSEEDocumentExists for doctitle
         tell application "SubEthaEdit"
             if exists document named doctitle then
                 document named doctitle
             else
                 make new document with properties {name:doctitle}
             end if
         end tell
     end ensureSEEDocumentExists
     
     to replaceContents for seeDoc by newContents
         tell application "SubEthaEdit"
             set the contents of seeDoc to newContents
             clear change marks of seeDoc
             try
                 set modified of seeDoc to false
             end try
         end tell
     end replaceContents
     
     to extendContents for seeDoc by moreContents
         tell application "SubEthaEdit"
             if "" is not equal to the contents of the last paragraph of seeDoc then
                 set the contents of the last insertion point of the last paragraph of seeDoc to return
             end if
             set the contents of the last insertion point of the last paragraph of seeDoc to moreContents
             clear change marks of seeDoc
             try
                 set modified of seeDoc to false
             end try
         end tell
     end extendContents
 ASCPT
 

A Bit More on AppleScript and stdin

2008-04-12T08:48:00.001-07:00

I can get even simpler than the earlier script to pass stdin to an AppleScript embedding within a shell script. Consider this:

#!/bin/sh
 
 /usr/bin/osascript > /dev/null <<ASCPT
     set stdinText to "$(cat | sed -e 's/\\/\\\\/g' -e 's/\"/\\\"/g')"
     tell application "TextEdit"
         activate
         make new document with properties {text:stdinText}
     end tell
 ASCPT
 

No need for the temporary file anymore, just use cat from within the AppleScript portion. The result of cat needs to be piped through sed in order to prevent problems with quoting, with a bunch of backslashes to get the special characters right. This may need further tweaking.

I ran into this when I tried using the earlier script for showing the results of the various scripts for the SubEthaEdit LaTeX mode. The earlier script failed when used for showing the results of bibtex. For some reason that I've not been able to work out, the AppleScript Standard Additions are no longer reached and the call to do shell script fails. The new approach works, so far at least.

AppleScript, Shell, and stdin

2008-04-10T11:00:00.001-07:00

AppleScript can be called from shell scripts, effectively giving access to Mac OS X applications from the underlying Unix tools. The AppleScript is embedded into the shell script as a here document, and invoked using osascript. I've given an example of this approach before.

However, there is something that I've never really been clear on. How does the AppleScript portion of the shell script read from stdin? Standard I/O is fundamental to Unix programming, so it is essential that AppleScript be able to access stdin. Searching with Google hasn't enlightened me. There seems to be no useful parallel with stdout, which works as you'd hope, with osascript writing transparently to it.

After a bit of thinking, I came up with this:

#!/bin/sh
 
 STANDIN=$(mktemp /tmp/seereport.XXXXXXXXXXXX) || exit 1
 cat > $STANDIN
 
 /usr/bin/osascript > /dev/null <<ASCPT
     set stdinText to do shell script "cat $STANDIN"
     tell application "TextEdit"
         activate
         make new document with properties {text:stdinText}
     end tell
 ASCPT
 
 trap 'rm -f $STANDIN' EXIT
 
 

I write stdin to a temporary file using cat, then read it back out in the AppleScript portion, again using cat. As an example, I just open a new TextEdit document with the text from stdin as its contents.

Saving this as minimal.sh, I can then create a new TextEdit document from the shell with:

echo hello world | ./minimal.sh

This works, so I've managed to get at stdin. It seems pretty roundabout though. Is there a better or recommended approach?

Update: I suppose it is worth mentioning that one could use pbcopy and pbpaste to avoid using the temporary file. However, doing so modifies the clipboard, so I prefer the approach shown.

Further, it would be possible to read the temp file using AppleScript commands, instead of calling cat with do shell script. That's too fiddly for a minimal example. Beyond that, I don't see much point to it, since I'd do any processing in the shell, just using AppleScript to pass the text to an application. I can't think of any applications where a stream approach would buy us anything.

Update: Modified example of using minimal.sh to actually use minimal.sh!

SEEing LaTeX 28: Some Critical Notes on the 'see' Command Line Tool

2008-03-23T12:17:00.001-07:00

I've used the see command line tool to relate a LaTeX previewer to SubEthaEdit using pdfsync. Unfortunately, I've become aware that the natural usage of see has a definite problem. Plugging in see and -g %line "%file" for the "PDFSync support" preferences in Skim doesn't work quite the way one would hope.

With that usage, which I'd consider to be the most natural, you wind up producing a new see process each time you command-click in Skim to switch over to SubEthaEdit. That process, due to shortcomings in the design of see, will hang around until you close the document in SubEthaEdit. Of course, if you're using SubEthaEdit and Skim together like that, you're in the middle of editing a LaTeX document, so you're not likely to be closing the document very promptly. Do that enough, and you could consume all the user processes allowed by Mac OS X. If you've not experience running out of user processes, let's just describe it as Not Fun.

I don't think it is likely to be a big problem, especially with the more friendly process limits in Mac OS X Leopard, but it is definitely a real problem. The problem gets compounded with each new usage of see for integrating external applications or reporting from a mode script. End result is that some caution is warranted when using see, and it should be omitted in favor of another approach, if necessary. Personally, I'm still using it.

I've requested an enhancement to see consisting of a 'quiet' mode, like seen in, e.g., grep. That would eliminate the issue, and, as a side benefit, considerably simplify using see from AppleScripts for SubEthaEdit modes. Let's hope the Coding Monkeys act on it.

SEEing LaTeX 27: End in Sight

2008-02-17T09:12:00.001-08:00

The activity here today, after a few weeks of silence, reflects that I've finished the documentation. I've spent some time earlier today getting everything ready. Finally, I am very pleased to announce that you can download the mode.

What comes next? That depends on you! Download it, try it out, and let me know about any problems, either by posting a comment or by sending me some email.

Update: The scripts developed in this series have been incorporated into the LaTeX mode distributed with SubEthaEdit. It should not normally be necessary to download the mode from the link given here, but I'll leave the link active in case anyone sees a need for my version.

Update 2: With SubEthaEdit 3.5, the LaTeX mode linked here is out of date, not supporting folding. You should not use it without good reason, as the mode that comes with SEE has everying this version does and more. I will leave the mode available for now, but I don't see much use for it.

SEEing LaTeX 26: Sample Environment Settings

2008-02-17T06:12:00.001-08:00

As a starting point for customizing the LaTeX mode, here are some sample environment settings:

<?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
 <plist version="1.0">
 <dict>
     <key>BIBINPUTS</key>
     <string>"$HOME/Library/texmf/bibtex/bib"</string>
     <key>SEE_BIBTEX</key>
     <string>'bibtex "${FILE%.tex}" | open -f -a SubEthaEdit'</string>
     <key>SEE_LATEX_CLEANUP</key>
     <string>'latexmk -C "$FILE"'</string>
     <key>SEE_LATEX_COMMENT</key>
     <string>'"$SEE_MODE_RESOURCES"/bin/comment.sh "% "'</string>
     <key>SEE_LATEX_COMPILER</key>
     <string>'latexmk -pdf -quiet "$FILE"'</string>
     <key>SEE_LATEX_COMPILEVIEWER</key>
     <string>'open -a Skim "$PRODUCT"'</string>
     <key>SEE_LATEX_PRODUCT_TYPE</key>
     <string>pdf</string>
     <key>SEE_LATEX_VIEWER</key>
     <string>'export __CF_USER_TEXT_ENCODING=0x1F5:0:0; /Applications/Skim.app/Contents/SharedSupport/displayline $LINE "$PRODUCT"'</string>
 </dict>
 </plist>
 

SEEing LaTeX 25: Putting Things in Order

2008-01-07T13:02:00.001-08:00

As noted below, the entries in the LaTeX mode menu for SubEthaEdit need to be put into a sensible order. After some thinking, I decided that there are really three groups of scripts: scripts for interacting with the LaTeX system, scripts for simplifying typing, and a script for interacting with the LaTeX mode itself.

Ideally, we'd put the three into groups divided by a horizontal rule. Unfortunately, SEE does not (yet) allow dividing lines to be inserted into the menu. Regardless, let's organize the scripts into the three groups, and - with one exception- just alphabetize within the groups. The exception is for the "Typeset and View" menu item; it strikes me as natural to put that first in the list.

End result is a menu ordered as:

Typeset and View
Clean Up Auxiliary Files
Run BibTeX
View

Complete Citation
Inline Math
Insert Environment...
Un/Comment Selected Lines

Customize Mode...

Although I've broken the groups apart with spacing, they'll just run together in the menu. Perhaps a later version of SEE will allow some more structure to be added.

One final change is that I've renamed the "Mode Environment..." menu item to "Customize Mode...". The two distinct meanings of "environment" struck me as confusing, and "Insert Environment..." is simply too appropriate for LaTeX to change.

I'll add another entry to the menu as soon as possible. It will go into the third group, with a title something like "Mode Help". I still have to write the mode help, first.

SEEing LaTeX 24: For Completeness, BibTeX

2008-01-06T12:06:00.001-08:00

I guess the right solution is to add a menu item for running bibtex, and that's it. Since I don't use makeindex myself, I'll leave that aside, unless someone wants to contribute scripts or just examples of use. I'd guess that the scripts would be easy enough, just adapt the ones I'll present for BibTeX.

Here's the AppleScript:

checkSaveStatus without updating
set bibScript to join of {modeEnvironment(), quotedForm for "$SEE_MODE_RESOURCES/bin/runbibtex.sh", quotedForm for documentPath()} by space
do shell script bibScript

on seescriptsettings()
return {displayName:"Run BibTeX"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

And here's the shell script that it calls:

#!/bin/sh
 
 #$Id: runbibtex.sh,v 1.1 2008/01/06 19:09:49 mjb Exp $
 
 PATH="$PATH:/usr/texbin:/usr/local/bin"
 export PATH
 
 BIBTEX=${SEE_BIBTEX:-'bibtex "$(basename $FILE .tex)"'}
 FILE="$(basename "$1")"
 DIRNAME="$(dirname "$1")"
 
 cd "$DIRNAME"
 eval $BIBTEX 
 

SEEing LaTeX 23: Am I Done?

2008-01-06T09:40:00.001-08:00

Last July, I listed some desirable features for the LaTeX mode for SubEthaEdit. The mentioned features were integrating with a PDF viewer using pdfsync, enabling insertion of citation keys in bibtex format, allow typesetting by calling pdflatex from within SEE, cleaning up auxiliary files, and commenting out selected lines. I also mentioned that it might be nice, if inessential, to be able to insert environments and formatting.

I've accomplished all of that. As well, I have introduced a mechanism for customizing the shell script environment for the mode and have developed a set of AppleScript handlers useful for scripting SEE. I'm quite pleased with how all of that has worked out. Not only do I now have a LaTeX mode that covers my main needs, but I've got a solid foundation on which I - and hopefully others! - can build support scripts for other modes.

That said, the LaTeX mode is not quite finished. It's clear that I should write some documentation, and add a "Mode Help" menu item. I should also better document SubEthaEditTools. Beyond that, there are two additional tasks.

First, the scripts in the LaTeX mode menu should be put into some sort of reasonable order, rather than the haphazard order that currently is there. The items appear in order based on the names of the scripts, which can differ from the entry in the menu. What is the right order to use?

Second, I've omitted some important elements of a LaTeX system, because latexmk handles them for me. For example, I have not provided a way to run bibtex. Should additional elements be added? If so, which? bibtex? makeindex? Something else?

SEEing LaTeX 22: Inline Math

2008-01-05T09:25:00.001-08:00

Environments aren't the only thing common LaTeX constructs that are awkward to type. The delimiters for inline math are pretty awkward, too. Let's add those, too:

set mathText to selectionText()
set wrappedText to "\$ " & mathText & " \$"
setSelectionText to wrappedText
if (length of mathText) equals 0 then
    set {startChar, nextChar} to selectionRange without extendingFront or extendingEnd
    setSelectionRange to (startChar + length of mathText + 3)
end if

on seescriptsettings()
    {displayName:"Inline Math", keyboardShortcut:"@~^m", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

SEEing LaTeX 21: Inserting Environments

2008-01-05T09:05:00.001-08:00

Environments again? Yes, but this time we're going to look at environments in LaTeX, not the shell environment. Environments are a bit of a pain to type, but the repetitive structure makes them suitable to automation: we just get the name of the environment, and then have a more-or-less standard form:

\begin{environmentname}
body
\end{environmentname}

Indentation of the body can be a little unclear. For example, I usually indent equation environments, but would never dream of indenting the document environment.

To add environment insertion into the LaTeX mode for SubEthaEdit, we begin by getting the environment name using display dialog:

try
display dialog "Enter environment name:" with title "Insert Environment" default answer "equation"
on error number -128 -- user canceled
return
end try
set envName to text returned of result

The try block is to handle when the user cancels instead of entering and environment name. Somewhat arbitrarily, I set the default answer to be "equation", since my guess is that equations are probably the most common environment.

After that, there is just some fiddly work getting the formatting of the environment correct. It needs to intelligently insert newlines and tabs to keep the document readable. As well, something needs to be done with the selection text. There are two possibilities as I see it: (1) treat the selection as the name of the environment and (2) treat the selection as the body of the environment. I went with the latter. Finally, it would be nice to place the insertion point somewhere reasonable; I think it works nicely at the end of the body, especially since the insertion point is positioned to start typing immediately if the body is empty. Putting it all together, we have:

set {startSelected, nextSelected} to selectionRange without extendingFront or extendingEnd
set {startExtended, nextExtended} to selectionRange with extendingFront and extendingEnd

set prefix to selectByComparing(startSelected, startExtended, "", "\n")
set suffix to selectByComparing(nextSelected, nextExtended, "", "\n")
set indent to selectByComparing(startSelected, nextSelected, "\t", "")

set beforeInsertion to (join of {prefix, "\\begin{", envName, "}\n", indent, selectionText()} by "")
set afterInsertion to (join of {suffix, "\\end{", envName, "}\n" } by "")

setSelectionText to (beforeInsertion & afterInsertion)
setSelectionRange to startSelected + (count of beforeInsertion) - 1 + (count of suffix)

Note that I've made repeated use of a convenience function:

to selectByComparing(val1, val2, sameVal, diffVal)
    if val1 equals val2 then
        sameVal
    else
        diffVal
    end if
end selectByComparing

The selectByComparing handler is not part of SubEthaEditTools - maybe it should be?

That's all there is to it, apart from the boilerplate:

on seescriptsettings()
{displayName:"Insert Environment...", keyboardShortcut:"@^e", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

Update: There is an interesting possibility for adjusting the default environment. I had used "equation" as the default, but it was pretty arbitrary. Another approach would be to just repeat whichever environment was last given. We just add a property to the script that holds the default environment, use its value in making the dialog, and update the property based on the dialog result:

property defaultEnvironment: "equation"

try
display dialog "Enter environment name:" with title "Insert Environment" default answer defaultEnvironment
on error number -128 -- user canceled
return
end try
set envName to text returned of result
set defaultEnvironment to envName

Is this actually a good idea? or will it just be annoying? Hard to say without using it, so I guess I'll try it out for a while.

SEEing LaTeX 20: Getting Citations Right

2008-01-04T12:38:00.001-08:00

I started enhancing the LaTeX mode for SubEthaEdit by looking at citations. The original approach, based on just using the input manager supplied with BibDesk, proved to be unsatisfactory. Let's fix that now.

The general approach seems clear enough. We need to determine a search term based on the cursor position in the LaTeX document, pass that to BibDesk to get matching documents, let the user pick which documents are relevant, format the selected documents, and insert the result into the document. Using SubEthaEditTools our SEE scripting abstraction layer, it proves to be fairly straightforward.

The first step, determining the search term, is probably the trickiest. What I envision is that the user can enter a partial citation and have it finished by the script. Thus, we'll need to examine the text immediately before the insertion point and determine a partial citation key. We should not cross an open brace "{", as we don't want to include the macro. Additionally, we shouldn't cross a comma, since we might be looking at multiple citations within one macro. Let's not check the calling macro; this allows the completion to be invoked at inappropriate points, but also allows the completion to be invoked for user-defined macros.

One final issue is what we do when there is a range of text selected. Since the default behavior should be to have no text selected, we should treat a non-empty selection as meaningful. Let's take it to mean that the search should be constrained to only include the selected text. Therefore, we determine a search term based either on the selected text or all the text on the line preceding the insertion point.

Putting all that together, I came up with:

set {startChar, nextChar} to selectionRange without extendingFront or extendingEnd
if startChar equals nextChar then
    -- empty selection, try the whole line
    set selectionContents to extendedSelectionText with extendingFront without extendingEnd
else
    set selectionContents to selectionText()
end if
set macroArgument to the last item of the (tokens of the selectionContents between "{")
set searchTerm to the last item of the (tokens of the macroArgument between ",")

I've used a couple of the new handlers from SubEthaEditTools. These are hopefully self-explanatory, but check the implementation in case they are not.

Using the two tokens calls, we obtain a partial citation key. We pass that to BibDesk:

tell application "BibDesk"
set citeMatches to search for searchTerm with for completion
end tell

The ungrammatical with for completion will give us a list of cite keys, not just document titles. Each completion is given as a string containing both the cite key and some document information, separated by a " % " string.

We next present the list of completions to the user, in order to narrow the list down to just the appropriate citations. To present a list, we use choose from list from the AppleScript StandardAdditions. There are three cases worth considering. First, if there is only one publication, we can select it by default, so the user just confirms whether it is correct. Second, if there are multiple possible publications, we just show the list, declining to guess. Finally, if there aren't any matches, we just inform the user with display alert; in this case, we'll set the list of publications to the empty list. If there aren't any publications returned, that means the user canceled, so we should just exit and leave the document unchanged. Putting it all together, we arrive at:

if (count of citeMatches) equals 1 then
    choose from list citeMatches with title "Citation Matches" with prompt "One matching publication:" default items citeMatches
    set pubs to result
else if (count of citeMatches) > 1
    choose from list citeMatches with title "Citation Matches" with prompt "Please select publications:" with multiple selections allowed
    set pubs to result
else
    display alert "No matches found for partial citation \"" & searchTerm & "\""
    set pubs to {}
end if

if (count of pubs) equals 0 then
    -- user canceled, do nothing
    return
end

We now have a non-empty list of completions. We need to split those apart to get the cite keys, then join the cite keys with commas. I used awk:

set citation to shellTransform of (join of pubs by "\n") for "" through "awk -F' % ' 'NR == 1 { printf(\"%s\", $1) } NR > 1 { printf(\",%s\", $1) }'" without alteringLineEndings

Pretty ugly! Let's reformat the core of the awk program to make it clearer:

NR == 1 { printf("%s", $1) } 
 NR > 1 { printf(",%s", $1) }
 

In this form, it's clear enough, keeping in mind that we use " % " as the field separator: we just print out the first field, corresponding to the cite key, with the first record treated specially to get the number of commas right.

Finally, we insert the formatted citation back into the document. I also move the insertion point to the end of the formatted citation, as a typing convenience. I had considered closing the brace for the citation macro, but that would make multi-citation lists more awkward, so decided against it. This is none too difficult:

setSelectionRange to {nextChar - (length of searchTerm), nextChar - 1}
setSelectionText to citation
setSelectionRange to (nextChar - (length of searchTerm) + (length of citation))

Again, I've used new handlers from the SubEthaEditTools library.

Putting it all together, and adding a seescriptsettings handler to integrate it into SEE, we get:

-- $Id: BibDeskCompletions.applescript,v 1.2 2008/01/04 18:40:16 mjb Exp mjb $

(*
Need to figure out the search term. Treat a selection as meaning to constrain the search
term to lie within the selection, and an empty selection as meaning to get the search term
from the preceding text on the line. We don't cross an opening brace, so that the search term
comes from a call to a macro. However, we don't check to see if the macro is one of the
standard citation macros, since we do want to allow user macros.
*)
set {startChar, nextChar} to selectionRange without extendingFront or extendingEnd
if startChar equals nextChar then
    -- empty selection, try the whole line
    set selectionContents to extendedSelectionText with extendingFront without extendingEnd
else
    set selectionContents to selectionText()
end if
set macroArgument to the last item of the (tokens of the selectionContents between "{")
set searchTerm to the last item of the (tokens of the macroArgument between ",")

tell application "BibDesk"
    set citeMatches to search for searchTerm with for completion
end tell

-- get list of publications, customizing user interaction based on number of matches
if (count of citeMatches) equals 1 then
    choose from list citeMatches with title "Citation Matches" with prompt "One matching publication:" default items citeMatches
    set pubs to result
else if (count of citeMatches) > 1
    choose from list citeMatches with title "Citation Matches" with prompt "Please select publications:" with multiple selections allowed
    set pubs to result
else
    display alert "No matches found for partial citation \"" & searchTerm & "\""
    set pubs to {}
end if

if (count of pubs) equals 0 then
    -- user canceled, do nothing
    return
end

(*
At this point, there is a non-empty list of matches, which replaces the search term. By
construction, the search term always immediately precedes the end of the selection.
Call out to the shell to format the publication list into a LaTeX citation, insert the citation,
and then move the insertion point just after the citation.
*)
set citation to shellTransform of (join of pubs by "\n") for "" through "awk -F' % ' 'NR == 1 { printf(\"%s\", $1) } NR > 1 { printf(\",%s\", $1) }'" without alteringLineEndings
setSelectionRange to {nextChar - (length of searchTerm), nextChar - 1}
setSelectionText to citation
setSelectionRange to (nextChar - (length of searchTerm) + (length of citation))

on seescriptsettings()
    {displayName:"Complete Citation", shortDisplayName:"Citation", keyboardShortcut:"@^j", toolbarIcon:"ToolbarBibDesk.png", inDefaultToolbar:"yes", toolbarTooltip:"Complete citation using BibDesk", inContextMenu:"yes"}
end seescriptsettings

include(`SubEthaEditTools.applescript')

SEEing LaTeX 19: SubEthaEditTools, or, Eating My Own Dogfood

2008-01-04T12:32:00.001-08:00

Let's take the arguments in the last three posts seriously. We move all of the handlers developed in the course of the SEEing LaTeX series into a separate file called SubEthaEditTools.applescript, and rewrite all of the AppleScripts to include the file using m4. While we're at it, we also add some more handlers to the SubEthaEditTools to try to make a more complete set, with the goal being to write scripts for the mode by using the handlers without directly interacting with SubEthaEdit in the AppleScript.

This requires relatively minor changes to the makefile I've been using, but nothing too serious. Also, I construct SubEthaEditTools itself using m4, with more general purpose handlers for, e.g., string manipulation placed in their own files. With the more modular structure, some additional minor rewrites of some handlers seems appropriate, which do not warrant specific comment.

A concept that is made explicit in the SubEthaEditTools is that of the extended selection. That is, the selection modified either so that the beginning is extended to the start of the first line of the selection, or to the end of the last line or the selection, or both. The extended selection has appeared implicitly several times, so it seems worthwhile to make it explicit. Further, there are now handlers both for extending the selection (forward or backward) and for referring to the extended selection without modifying the actual selection.

Update: I've replaced the quotedForm handler with a doubleQuotedForm handler, to help prevent confusion with the quoted form of action for AppleScript strings. I've also worked in a few usages of quoted form of in the various scripts for the LaTeX mode (I didn't know about quoted form of until recently).

Also, I've added a license. It's an MIT-style license, so should be suitably permissive for use by others.

Update 2 (2008/04/23): I've added a few more handlers. Specifically, modeSetting, selectionIsEmpty, and documentIsAvailable.

-- SubEthaEdit Tools

(*
Copyright (c) 2008, Michael J. Barber

Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject
to the following conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR
ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF
CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

*)

-- Environment management

on modeEnvironment()
    join of {"export __CF_USER_TEXT_ENCODING=0x1F5:0x8000100:0x8000100;", "export SEE_MODE_RESOURCES=", doubleQuotedForm for modeResources(), "; ", readEnvironment out of environmentFilePath()} by ""
end modeEnvironment

on modeSetting for envVar
    try
        tell application "System Events"
            tell property list file (my environmentFilePath())
                get value of the property list item envVar
            end tell
        end tell
    on error
        missing value
    end try
end modeEnvironmentSetting

to openEnvironmentSettings()
    openEnvironment at environmentFilePath() with settingDefaultEnvironment
end openEnvironment

on environmentFilePath()
    tell application "SubEthaEdit" to set modeName to name of the mode of the front document
     join of {path to preferences from user domain as string, "de.codingmonkeys.SubEthaEdit.", modeName, "_environment.plist"} by ""
end environmentFileName

-- Manipulation of document text

on documentText()
    tell the front document of application "SubEthaEdit" to get the contents
end documentText

on selectionIsEmpty()
    tell the front document of application "SubEthaEdit" to get the length of the selection
    result is equal to 0
end selectionIsEmpty

to completeSelectedLines()
    extendSelection with extendingFront and extendingEnd
end completeSelectedLines

on selectionText()
    tell the front document of application "SubEthaEdit" to get the contents of the selection
end selectionText

to setSelectionText to newText
    tell application "SubEthaEdit" to set the contents of the selection of the front document to the newText
end setSelectionText

on selectionRange given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    tell the front document of application "SubEthaEdit"
        if shouldExtendFront and shouldExtendEnd then
            get {startCharacterIndex of the first paragraph of the selection, nextCharacterIndex of the last paragraph of the selection}
        else if shouldExtendFront then
            get {startCharacterIndex of the first paragraph of the selection, nextCharacterIndex of the selection}
        else if shouldExtendEnd then
            get {startCharacterIndex of the selection, nextCharacterIndex of the last paragraph of the selection}
        else
            get {startCharacterIndex of the selection, nextCharacterIndex of the selection}
        end if
    end tell
end selectionRange

to setSelectionRange to newRange
    tell the front document of application "SubEthaEdit"
        set selection to newRange
    end tell
end setSelectionRange

on extendedSelectionText given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    set {startChar, nextChar} to selectionRange given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    tell the front document of application "SubEthaEdit"
        get the contents of characters startChar through (nextChar - 1) as text
    end tell
end extendedSelectionText

to extendSelection given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd
    set {startChar, nextChar} to (selectionRange given extendingFront:shouldExtendFront, extendingEnd:shouldExtendEnd)
    setSelectionRange to {startChar, nextChar-1}
end extendSelection

-- Manipulation of document properties

to checkSaveStatus given updating:shouldSave
    tell application "SubEthaEdit"
        if not (exists path of front document) then
            error "You have to save the document first"
        end if
        if shouldSave and (modified of front document) then
            try
                save front document
            end try
        end if
    end tell
end checkSaveStatus

on documentIsAvailable()
    tell application "SubEthaEdit" to get the count of the documents
    result > 0
end documentIsAvailable

to requireNewlineAtEOF()
    tell the front document of application "SubEthaEdit"
        if "" is equal to the contents of the last paragraph then
            -- final line terminated, do nothing
        else
            set the contents of the last insertion point of the last paragraph to return
        end if
    end tell
end requireNewlineAtEOF

on documentPath()
    tell application "SubEthaEdit" to get the path of the front document
end documentPath

on documentLine()
    tell application "SubEthaEdit" to get the startLineNumber of selection of front document
end documentLine

on modeResources()
    tell application "SubEthaEdit" to get the resource path of the mode of the front document
end modeResources

-- String Utilities

on replacement of oldDelim by newDelim for sourceString
    return join of (tokens of sourceString between oldDelim) by newDelim
end replacement

on tokens of str between delimiters
    set oldTIDs to text item delimiters of AppleScript
    set text item delimiters of AppleScript to delimiters
    set strtoks to text items of str
    set text item delimiters of AppleScript to oldTIDs
    return strtoks
end tokens

on join of tokenList by delimiter
    set oldTIDs to text item delimiters of AppleScript
    set text item delimiters of AppleScript to delimiter
    set joinedString to tokenList as string
    set text item delimiters of AppleScript to oldTIDs
    return joinedString
end join

on doubleQuotedForm for baseString
    quote & baseString & quote
end doubleQuotedForm

on shellTransform of inText for envString through pipeline given alteringLineEndings:altEnds
    set shellscript to join of {envString, "pbpaste", "|", pipeline} by space
    set the oldClipboard to the clipboard
    set the clipboard to the inText
    try
        set shellresponse to do shell script shellscript altering line endings altEnds
    on error errMsg number errNum from badObject
        set the clipboard to the oldClipboard
        error errMsg number errNum from badObject
    end try
    set the clipboard to the oldClipboard
    shellresponse
end shellTransform

-- Handling of environment settings using a plist file

to writeDefaultEnvironment at envPath
    set savedClipboard to the clipboard
    set the clipboard to "<?xml version=\"1.0\" encoding=\"UTF-8\"?>
<!DOCTYPE plist PUBLIC \"-//Apple Computer//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\">
<plist version=\"1.0\">
<dict/>
</plist>"
    try
        do shell script "pbpaste > " & (POSIX path of envPath)
    on error errMsg number errNum from badObject
        set the clipboard to the savedClipboard
        error errMsg number errNum from badObject
    end try
    set the clipboard to the savedClipboard
end writeDefaultEnvironment

to openEnvironment at envFilePath given settingDefaultEnvironment:shouldSetDefault
    tell application "System Events"
        if not exists file envFilePath
            if shouldSetDefault
                my writeDefaultEnvironment at envFilePath
            else
                error ("Can't get environment file " & quote & envFilePath & quote) number -1728
            end if
        end if
        open file envFilePath
    end tell
end openEnvironment

to readEnvironment out of plist
    readListPair out of plist
    environmentString from result
end readEnvironment

to readListPair out of plist
    tell application "System Events"
        if exists file plist then
            tell property list file plist
                get {name, value} of every property list item
            end tell
        else
            {{}, {}}
        end if
    end tell
end readPlist

on environmentString from keyValueListPair
    set {plistKeys, plistValues} to keyValueListPair
    set accumulator to {}
    set oldTIDs to text item delimiters of AppleScript
    set text item delimiters of AppleScript to ""
    repeat with i from 1 to number of items in plistKeys
        set tokens to {"export ", item i of plistKeys, "=", item i of plistValues, ";"}
        copy (tokens as string) to the end of the accumulator
    end repeat
    set AppleScript's text item delimiters to space
    set envString to accumulator as string
    set AppleScript's text item delimiters to oldTIDs
    envString
end environmentString