A tip of the iceberg

This blog has always been about the non-work component of my life: e.g. marathon running, cycling or working out in general. Some topics are obviously terribly geeky and of limited general interest, e.g. table tennis or spherical panorama photography.

Recently, I've been mainly working and working out. On the work side, one of the things I sometimes do is write computer programs that implement (mainly scientific) theories that I'm interested in. In particular, I'm interested in theories of grammar. Note that I keep mentioning this word "interesting" again. I suppose I'm kinda lucky in that in some sense there is a component of my work that would also be a hobby of mine. The programming side does not necessarily stem from practicality: I simply enjoy making these theories become tangible. I am a professional computer programmer. But I like to think it's my hobby as well.

As with hobbies, let me quote Alan Parsons Project 's Limelight and say "maybe the prize is small," but the time and effort that goes into parts of a programming project is sometimes ridiculously disproportionate.

To illustrate this point, let's take a simple example. I'm currently rewriting a program maybe for the 4th or 5th time in its lifetime. I've gone from hosting the program initially on a Symbolics 3650 Lisp machine (with microcode assist for Prolog), then a Sun Sparc workstation, then Apple's Powerbook (PPC), and finally to a Macbook (Intel). (I apologize in advance for the super-geekiness in this post. In an attempt to not lose the plot, I've put the most dire and geekiest parts in separate paragraphs with small fonts.)

As an aside, the Symbolics machine (see here) provided a programming environment that is truly unequaled. How can you match a system that provided seamless full source clickability from your application, through the windowing and underlying operating system, all written in Lisp? Or a keyboard that provided Hyper, Super and Meta as well as a Control key?

The core of my program implementing the theory I was interested in hasn't changed very much apart from porting considerations. It's mostly written in Prolog, a logic programming language that is way past its heyday, plus bits in raw C for efficiency. (Remember, computers used to be so much slower.)

Over the years, I've had to port over various flavors and wrestle with differing foreign language interfaces, from Quintus (commercial), to SICS's Sicstus (semi-commercial, restrictions on runtime environments) finally to University of Amsterdam's SWI (freeware). Freeware at last! These days, I've also taken out the C bits for portability. As anyone can tell you, dynamic library linking is simply no fun when OS libraries are patched and released all the time.

The part of the program that has changed greatly has been the user interface component. Unfortunately, it has been completely rewritten from scratch a number of times. That came with the lack of standardization. The latest rewrite, the subject of this blog post, uses the Tcl/Tk toolkit with native Aqua Mac OS X support. Here is a sample of what it looks like now:

I'm a design minimalist at heart when it comes to user interfaces (as well as other things). I believe user interface complexity can be overwhelming; it's best hidden from view and enabled only when necessary to the task. Note that there is only one button visible ("clear trees"). In the relatively austere-looking window above, there are additional menus and detailed information hidden from view that can be displayed by clicking on or passing the mouse through relevant display objects. In this blog post, I'll concentrate on programming the behavior of a single detail, the blue sideways tab near the top-right of the window.

Tcl stands for Tool Control Language, a simple interpreted programming language. Tk is a graphical user interface toolkit that can be used with Tcl. In contrast to native toolkits from Microsoft or Apple, Tcl/Tk is multi-platform and well supported natively. Moreover, due to its interpreted nature, you don't need to recompile or relink your application each time you make changes. There are other viable choices, e.g. Java and Swing from Sun Microsystems (before it was bought out by Oracle), but Tcl/Tk has that spare minimalist look to its programs that appeals to me.

Initially, the user interface was written in Lisp, effortlessly borrowing the tree layout and mouse clickability facilities that came with the Symbolics programming environment to render syntax trees. Then it was onto the X Window System on Unix. Remember X11R3 and X11R4? The C-based XView libraries plus Slingshot extensions was supported by Sun Microsystems and implemented the OpenLook interface guidelines. X11 for Mac OS X allowed recompiled executables to run on the Mac. However, generational changes in the X11 libraries as Mac OS X morphed over the years resulted in niggling lost functionality in the best case and random core dumps in the worst. Moreover, it didn't mesh well with the arguably slicker native Mac OS X graphical user interface.

A menu of commands (useful mostly when starting up or quitting) is hidden under that blue tab.

As a design minimalist, I don't want those commands unnecessarily cluttering up the user interface or inconsistently taking up valuable screen real estate.

But if you mouse over that tab, the tab extends out a little to the left, and the command menu pops up as shown here.

However, when the program is first launched, this design minimalism results in a seriously over-sparse window. See below.

It's not terribly clear how to proceed or what to press. So, as a helpful hint, I thought it'd useful to automatically pop that menu up at launch, as if the tab had been manually "moused over", and the user can see he/she should select something intrinsically useful from the menu like "load defaults", which in turn will lead to more relevant interface components materializing.

It's not really necessary to explain in detail how this programming is accomplished. (See below.) It's the idea behind it that matters; in other words, that hinting at startup might be helpful and appreciated by the user.

The way to elegantly pop up the menu without writing much additional code is simply to simulate moving the pointer over the blue tab. User interface code is event-driven. Therefore, we write code that "binds" or associate actions with specified user interface events.

For example, if the pointer moves over the blue tab, a logical <Enter> event is generated. If we have previously associated the menu with <Enter> for that tab, the menu will pop up reactively to moving the mouse (or finger on the trackpad) to the right location. So at startup, we pretend that the user has moused over that tab simply by posting <Enter> blue tab to the event queue and letting the user interface code do its work.

For those curious, here's what it looks like in Tcl/Tk:

1. label .f.msg.tab -image tab_blue -width 23 -height 32
2. bind .f.msg.tab {popup_tab %X %Y}

Line 1 says .f.msg.tab is a label displaying the blue tab, and line 2 says we've bound popup_tab to an <Enter> blue tab event. In our startup code, we post the event virtually by simply saying:

event generate .f.msg.tab -x 10 -y 10

(x and y are the screen coordinates.)

We would have nothing further to say except that, unexpectedly, the obvious code doesn't work. In particular, not only the menu doesn't automagically pop up, worse still, it generates an incomprehensible and untraceable error as shown below:

$ wish interface.tcl
popup_tab 622 87
bgerror failed to handle background error.
    Original error:
    Error in bgerror: wrong # args: should be "text id"
^C

(Here, wish is the name of the program that interprets Tcl programs. And interface.tcl is the file that contains the source code that implements the graphical user interface shown earlier.)

Never mind about the aesthetics, programs that bomb out aren't acceptably user-friendly. Like the tipping point of an avalanche, this error generates a cascading fury of activity by the programmer that only ends when the dust has settled and a new equilibrium has been reached. Well, for the programmer the choice is simply to give up now or concentrate furiously - lest you lose that train of thought - and doggedly follow the chain of inquiry to its logical conclusion and eventual solution, at which point 10pm has somehow morphed into 4am or 5am accompanied by blurry eyes and stiff, aching shoulders.

(And that's of course if you're fortunate. If you're not, you're out one night's sleep and the problem still stands.)

I'd like to think it's not necessarily obsessive compulsive behavior, rather it's the nature of the beast that forces these debilitating all-night debugging sessions. Having been completely immersed in the details of chasing the bug down for hours, a lot of short-term memory has been committed to the task at hand. You can't just simply pull the plug or switch off in the middle of it all; it's remarkable how many of those details committed to short-term memory will vanish in the dark of the night. Next morning, you could be faced with the overhead of considerable detective work to retrace and piece together again the current half-analyzed state of the puzzle. You could try to write down everything you've done so far before going to bed (to explain things to tomorrow's you), but it's far easier just to get on with it and burn that midnight oil.

The hunt for an explanation and a solution

So how do we track this one down? Well, for a start we have to be reasonably sure we haven't made a mistake in the code. We first confirm, that without the virtual event, the menu does pop up reliably. From the Tk toolkit's perspective, it's desirable that there should be no semantic distinction between a simulated and an actual event.

This example I've chosen is not as trivial as one might expect. The ability to post virtual events as if they were truly real is an important and valuable feature for user interface toolkits. For example, this would allow demos to be easily built. And seeing is understanding sometimes. The program could automatically show the user how to perform certain actions.

Understanding the behavior

The first clue to the puzzling behavior comes when I instruct the program to delay posting the virtual event by a few seconds, and I click to bring the program's window to the foreground before the event is posted. Ee bah gum, the menu pops up without the error message! Unfortunately, I can't really always ask the user to foreground the window quickly at startup before the virtual event posts; if he/she could grok that, they probably wouldn't need or necessarily appreciate the menu hint in the first place. Saying this is a feature not a bug would also be a serious cop-out. We are better than that (I hope)!

Going to the documentation

The next step is to look at the Tk documentation for generating virtual events:

The last line provides the clue we need to gain traction in our inquiry. It states "Certain events, such as key events, require that the window has focus to receive the event properly." Of course, this doesn't match the error message we received: Error in bgerror: wrong # args: should be "text id". And "receiving the event properly" is kinda wishy-washy and not properly defined. Plus a mouse or pointer event isn't really the same thing as pressing a key on the keyboard. But an experienced programmer will smell blood immediately.

The reason why this is a good clue because the program works when we delay the virtual event and click first to foreground the window, thereby giving it focus.

Going to the comp.lang.tcl.mac mailing list

Okay, we need to be able to write code that forces focus to be assigned to our program window as soon as the process is started. If we can do this before the virtual event is posted, we'll have no error message. Unfortunately, despite perusing the Tk documentation frantically, there appears to be no way to force Mac OS X to do this. Do we give up? No, of course not. A good programmer instinctively knows someone out there must have encountered and bitched about this problem. It'll just require some efficient and judicious application of the right search terms to ferret out the discussion. Let's go to the web then.

In the dark days before graphical user interfaces, Safari browsers and the World Web Web, there was Usenet. People could post questions and issues to a hierarchy of newgroups, the contents of which would be asynchronously transmitted on the Arpanet from host to host. Given the limits of storage back then, a typical hosts would retain a few weeks or months of these sometimes valuable discussions. Nowadays of course, Google is your friend, and all these discussions have achieved immortality, being permanently stored and publically available to all and sundry.

comp.lang (computer languages) is a sub-node in this hierarchy, and comp.lang.tcl is a subgroup that deals specifically with the Tcl language. And comp.lang.tcl.mac is a subgroup that deals with Tcl on the Mac platform.

Our instincts are correct. Someone named Steven back in 2009 asked the following relevant question on comp.lang.tcl.mac:

And someone gave an authoritative and informative reply:

The informative reply requires a bit of decoding though. tclCarbonProcesses extensions and a teapot?

The first thing a good programmer would do would be to check if these extensions might already be available on his computer. After all, Mac OS X already occupies several gigabytes on his hard drive. Unfortunately, as the following dialog shows, it isn't.

$ wish
% package require tclCarbonProcesses
can't find package tclCarbonProcesses
% package names
ttk::theme::classic http tcl::tommath tcltest ttk::theme::default Ttk ttk::theme::aqua msgcat Tcl ttk::theme::clam platform tile Tk ttk::theme::alt

Not to worry, we'll just have to download and install this tclCarbonProcesses extension package.

tclCarbonProcesses extensions and a teapot

This is where things start to get a bit hairy. I didn't have luck with the repository teapot.activestate.com. However, there is a wikipage for tclCarbonProcesses.

But the web link provided is broken. But the file tclCarbonProcesses.tcl seems to be all there. So we have the source. And it's written in Tcl.

But wait a moment, this is no ordinary Tcl program. tclCarbonProcesses appears to be a Critcl wrapper for Mac OS X Process Manager services. So we need critcl first. And of course, it's not installed on our computer. And what in heaven's name is critcl anyway?

A critcl wrapper

More googling:

Critcl, which tclCarbonProcesses.tcl requires, is an acronym for Compiled Runtime in Tcl. Very cute. It allows C code to be inline embedded in Tcl. I have a vague uneasiness on many fronts about this, but having come this far, let's press on.

Let me briefly explain the source of my unease. The reason why I'm using the Tcl/Tk toolkit is to achieve platform independence. And I've also tried to eliminate linking in specific C code for portability reasons as well. tclCarbonProcesses.tcl is both platform-specific (Mac OS X) and apparently embeds C code. Moreover it appears a bit kludgy and might not be supported or work in future editions of Mac OS X.

Furthermore, it appears from the critcl manpage that critcl is supplied as a Starkit. And we need something called Tclkit in turn to install and run it.

Tclkit

Are you still with me here gentle reader? To recap, I need to download and install Tclkit because critcl needs it. And I need to install critcl because tclCarbonProcesses.tcl needs it. And I need the tclCarbonProcesses package because it allows me to set focus for my program window. And I need to set focus because Tcl needs it to support virtual events without raising a bgerror.

At this point, even a brave and intrepid programmer might think it's prudent to reassess the situation. After all, experience tells him there could easily be trouble getting any of Tclkit, critcl and tclCarbonProcesses to install and work properly. And none of them can be skipped or omitted: all of them are required in order to get the programmatic focus he so desires.

(The programmer has also started referring to himself in the third person, rather than the first. It's okay. He is feeling somewhat detached from the real world by this time.)

Moreover, the programmer ruefully rubs his chin and notes that even if all of the packages are installed and working properly, and his program manages to grab focus via a command, the error might still be there. After all, who is to say there is no difference between focus achieved manually (via a mouse click) or programmatically? This is not some utopian environment in which events, real or imaginary, get to live harmoniously together. It's the real world of imperfect computer programming languages and flakey windowing systems. In other words, there is no guarantee the fix will work anyway.

But it's getting late in this almost poker-like game where the stakes keep getting raised. He realizes there are far too many single points of failure in this scenario. But he is in too deep to back out. He has wasted hours already. Fully committed, it's all or nothing.

He finds there are two version of the binary executable tclkit for darwin (Mac OS X) floating around on the internet. He optimistically installs version 2.

$ ls tcl*
tclkit-darwin-univ-aqua 2 tclkit-darwin-univ-aqua.gz

$ mv tclkit-darwin-univ-aqua\ 2 tclkit
$ chmod +x tclkit
$ mv tclkit ~/bin

$ which tclkit
/Users/sandiway/bin/tclkit

Next, he find there are two versions of critcl, critcl.kit.sh and critcl2.kit.sh. He downloads both and tests critcl2.

$ sh critcl2.kit.sh -test
exec = /Users/sandiway/bin/tclkit
prog = /Users/sandiway/Downloads/critcl2.kit.sh

As the above test indicates, critcl2 seems to pass a simple self-test. So he tries to process tclCarbonProcesses.tcl using critcl2:

$ sh critcl2.kit.sh tclCarbonProcesses.tcl
critcl2.kit error: No compiler found

It throws up an error message saying "No compiler found". This is strange. Since critcl allows C code to be inline embedded in Tcl, it must be the C compiler that it can't find. And the C compiler of choice is gcc.

C Compiler gcc

The programmer is starting to question his own sanity at this point. Shurely shome mishtake? He knows he has used gcc on this machine before. How come it's no longer there?

He investigates this rather dubious situation using those two powerful Unix commands which and locate.

$ which gcc
$ locate gcc

WARNING: The locate database (/var/db/locate.database) does not exist.
To create the database, run the following command:

sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.locate.plist

Please be aware that the database can take some time to generate; once
the database has been created, this message will no longer appear.

The command which turns up nothing so it's not in his executable path. locate is particularly unhelpful, claiming the hard drive needs to be indexed for it first to find things. The programmer demurs since he has a half-full 640GB hard drive in his laptop, and that would take a very long time indeed to index.

He finally sorta locates it in /Developer/usr/bin. And runs a simple check using the "-v" flag. Despite the verbiage reported below, he is not fooled by the self-test. In fact, he is genuinely and deeply troubled. He knows for a fact that he has installed Apple's Xcode and used gcc before. Why then has it stopped working?

$ /Developer/usr/bin/gcc -v
Using built-in specs.
Target: i686-apple-darwin10
Configured with: /var/tmp/gcc/gcc-5666.3~6/src/configure --disable-checking --enable-werror --prefix=/usr --mandir=/share/man --enable-languages=c,objc,c++,obj-c++ --program-transform-name=/^[cg][^.-]*$/s/$/-4.2/ --with-slibdir=/usr/lib --build=i686-apple-darwin10 --program-prefix=i686-apple-darwin10- --host=x86_64-apple-darwin10 --target=i686-apple-darwin10 --with-gxx-include-dir=/include/c++/4.2.1
Thread model: posix
gcc version 4.2.1 (Apple Inc. build 5666) (dot 3)

Again, he goes to the web. Yes! Someone has bitched about this exact problem.

Yes, just like the poor sod with the handle Lolmaniac, this programmer used to run Mac OS X 10.5 - codenamed Leopard - on this laptop. And like Lolmaniac, he had upgraded to 10.6 - aka Snow Leopard. So he needs now, in the middle of the night, to dig out his Snow Leopard installation dvd.

Snow Leopard

By the grace of some tremendous and most merciful deity, although he now longer can remember whether or what he had for dinner tonight, he remembers where he put his original Apple Snow Leopard disk.

Triumphantly, he inserts the dvd in his laptop drive, and half an hour and a few hundred megabytes later, Xcode for Snow Leopard, and therefore, gcc now live again on his computer. He notes with a disproportionate amount of satisfaction and glee that the which command now finds gcc on the most righteous path, i.e. in its anointed and proper location, the hallowed /usr/bin.

$ which gcc
/usr/bin/gcc

Update: well, the hit wasn't really just several hundred megabytes. Fact-checking the next day using du, I see that /Developer is 2.0 gigabytes large, though I confess to being unclear whether earlier versions remain or are included in that total...

$ du -s -h /Developer
2.0G /Developer

Plus of course, next day, Xcode wanted to download a 650MB update to all that..

Back to critcl

After that several hundred megabyte gigabyte sacrifice, the programmer returns to trying to run critcl on the tclCarbonProcesses package. The build fails and it crashes with an almightly thump of error messages.

$ sh critcl2.kit.sh -pkg tclCarbonProcesses.tcl
Target: universal-macosx
Source: tclCarbonProcesses.tcl
Tue Aug 09 02:59:51 MST 2011 - /Users/sandiway/Downloads/tclCarbonProcesses.tcl
gcc -c -arch i386 -arch ppc -isysroot $SDKROOT -mmacosx-version-min=$osxmin -DUSE_THREAD_ALLOC=1
-D_REENTRANT=1 -D_THREAD_SAFE=1 -DHAVE_PTHREAD_ATTR_SETSTACKSIZE=1 -DHAVE_READDIR_R=1
-DTCL_THREADS=1 -DUSE_TCL_STUBS -I/Users/sandiway/.critcl/universal-macosx -o /Users/sandiway/.critcl/universal-
macosx/v20_f082abebe39115d7e5eb1db8a2e0a5c9_pic.o /Users/sandiway/.critcl/universal-
macosx/v20_f082abebe39115d7e5eb1db8a2e0a5c9.c -O2 -DNDEBUG
cc1: error: missing argument to "-mmacosx-version-min="
cc1: error: missing argument to "-mmacosx-version-min="
lipo: can't figure out the architecture type of: /var/folders/G+/G+85fomYEnS8Eu68w9O4JU+++TI/-Tmp-//ccUVY2kt.out
ERROR while compiling code in /Users/sandiway/Downloads/tclCarbonProcesses.tcl:
child process exited abnormally

critcl build failed (/Users/sandiway/Downloads/tclCarbonProcesses.tcl)
Files left in /Users/sandiway/.critcl/universal-macosx

Normally, this single point of failure would end his efforts here tonight, but remember he downloaded two versions of critcl. He abandons critcl2, and fires up critcl version 1. Mercifully, it completes without emitting a single error message.

sh critcl.kit.sh -pkg tclCarbonProcesses.tcl
Source: tclCarbonProcesses.tcl
Library: tclCarbonProcesses.dylib
Package: /Users/sandiway/Downloads/lib/tclCarbonProcesses

As the above dialog indicates, he finally now has a dynamic link library called tclCarbonProcesses.dylib. This is pretty good news. He needs to test it though. But first, where should he put this library so the Tcl interpreter (wish) can automatically find and load it?

Diving through unfamiliar parts of the Tcl documentation, he finally locates the piece of information he is looking for in pkg_mkIndex, a Tcl command which he has never needed to invoke, and hopefully, never will.

The pkg_mkIndex documentation tells him there are four steps he must follow in order to make the Tcl command package require (that loads in optional packages) operate smoothly. But he is vastly experienced and not so easily sidetracked by wanton instructions (or so he believes). He finds the critical clue in the documentation of step 3:

It teases him like something terrible out of that bad Dan Brown novel "The DaVinci Code". In other words, it doesn't exactly tell him what he is looking for, but it tells him he can find out by inspecting the value of the Tcl variable $tcl_pkgPath:

$ wish
% puts $tcl_pkgPath
/System/Library/Frameworks/Tcl.framework/Versions/8.5/Resources/Scripts ~/Library/Tcl /Library/Tcl /System/Library/Tcl /System/Library/Tcl/8.5 ~/Library/Frameworks /Library/Frameworks /System/Library/Frameworks

Looking through the list of directories, and not wishing to customize any more than absolutely necessary, he settles for putting tclCarbonProcesses under ~/Library/Tcl:

$ pwd
/Users/sandiway/Downloads
$ cd lib/
$ ls
tclCarbonProcesses
$ mv tclCarbonProcesses/ ~/Library/Tcl/

And so it came to pass, the Tcl interpreter (wish) loaded tclCarbonProcesses when requested to do so, and it also managed to front the interpreter and assign it focus.

(You see, in the picture above the commands were entered using the Terminal window. Normally, the window you are interacting with, e.g. typing, will have focus. But notice that only the (blank) Wish window has colored decorations in the header. Those non-grayed decorations signals that it has focus.)

Hinting

The programmer decides to insert the setFrontProcess code into his graphical user interface program. The moment he has striven for through the long night has finally arrived. He fires up the program and it returns the same original stupid error message.

Undeterred, much like repeatedly wacking a misbehaving tv set that is on the fritz, he fires up the application again. And it works. And has continued to work ever since.

Now, gentle reader if you have made it so far, you must be wondering if it was all worth the trouble. Technically speaking, depending on how the application might be deployed - maybe not, but it affirmed this programmer that he still got what it takes. You see, during regular semester time, he plays professor and would never have a block of undistracted time large enough to tackle such a task.