[cells-gtk-devel] Re: Multithreading patch

[Peter Hildebrandt]

Here it is, finally.  Multithreading for cells-gtk on sbcl/linux.

http://www.washbear-network.de/peterblog/wp-content/uploads/2007/12/cells-gtk-threading-2.patch

(since it's too big for the list, I revived my blog and put stuff there.   
Check it out if you like: <http://www.washbear-network.de/peterblog>)

I built the patch against the current cvs (which is different from the  
tar.gz linked on the website, which is different from the asdf-install  
version).  Applications will work just as they used to (a littler better,  
hopefully, see below).

To run an application in the background, just use the new

(start-win 'app-class initargs)

which starts your application window in the background and returns right  
to the repl.  The return value is the freshly created instance of  
app-class.  That's it.

To initialize cells-gtk, there is the new (init-gtk).  It is threadsafe  
and makes sure that no important state is destroyed.  That is, you can  
call it while other windows are open in the background.  Don't do this  
with cells-gtk-init.  You will need to call init-gtk before you do  
anything to cells-gtk (it loads foreign libraries).  start-win takes care  
of that for you.

A couple details:

- To simulate the idea of a "main window" (thanks Kenny for pointing out  
that this is not something "natural") gtk-app now has the slot  
terminate-on-exit -- if this is set to true, all open windows will be  
closed when this is closed.

- (open-windows) gives you a list of, well, open windows

- You can close all windows by (close-all-windows) or (init-gtk  
:close-all-windows t)

- Calls from the repl can be wrapped in (with-gdk-threads ...), in theory  
this is the right thing to do.  In practice it mostly works if you don't.   
However, every now and then I broke my gdk this way and had to restart  
lisp.  In a real app, always wrap calls that might affect your window in  
(with-gdk-threads ...).

Multithreading and gtk is tricky.  Calls have to wrapped in the macro  
*exactly once* -- do it more than once, and you get a deadlock.  Do it  
less than once, and gtk might fail with an Xlib asynchronity.

For now, wrap everything at the highest level in with-gdk-threads, except  
for start-win.

- You can start the gtk thread manually by calling (start-gtk-main).   
start-win does that for you.

- The gtk thread can be terminated by calling (stop-gtk-main).  In order  
to use gtk again, you will have to restart lisp.  Normally this is utterly  
useless.  You might want it if you ship your app as a save-lisp-and-die  
image.  Maybe.


Here's a silly example of interactive UI development:

;; make a package
(defpackage :cgtk-user (:use :cl :cgtk :cells))
(in-package :cgtk-user)

;; start a blank app
(start-win 'gtk-app :title (c-in "My App") :kids (c-in nil)) ; it appears

;; bind it to a variable
(defparameter *app* *)

;; change its title
(setf (title *app*) "Application") ; the title changes

;; create a vbox to take the our widgets
(push (mk-vbox :kids (c-in nil)) (kids *app*))
(defparameter *vbox* (first *))

;; make a button
(push (mk-button :label "Click me" :on-clicked (callback (w e d) (print  
'you-clicked-me))) (kids *vbox*)) ; it shows up

;; make a counter
(push (mk-button :label (c-in "Me too") :on-clicked (let ((count 0))  
(callback (w e d) (print (incf count))))) (kids *vbox*))

;; relabel that button

(setf (label (first *)) "Counter")

;; close all windows
(close-all-windows)

That's all I can think of right now.  Let me if clarifications are needed.

Regards,
Peter


Some technical details:

- The way gtk-app used to handle restarts resulted in effect in a mutual  
recursion:  Closing a window resulted in signaling quit, which led to  
transfer of control out of gtk-main into lisp, which then exited, leaving  
the gtk-main task waiting for a return (that would never come).  Each time  
you ran an application, that stack would build up.

The previous way to unwind that stack upon exit, like

   (loop for i below (gtk-main-level)  do (gtk-main-quit))  ;; I had this  
one somehow

or

   (loop while (> (gtk-main-level) 0) do (gtk-main-quit))   ;; from cvs,  
results for me in endless loop, since gtk-main-level won't reduce for me  
by calling gtk-main-quit like this

This does not do the trick, since gtk-main-quit only works as expected for  
active gtk-main loops, not for stale ones (which you get after  
transferring control out of them by means of a signal or error).  In  
effect, I found no way to resume an open instance of gtk-main once a lisp  
callback has used error or signal to uncleanly jump out of it.  I believe  
that this could be the reason why the termination code in gtk-app.lisp  
needs several workarounds for different lisps:  something's not kosher  
here.

I rewrote gtk-app completely so that now in effect one instance of  
gtk-main runs continously.  When an application with :terminate-on-close  
is closed, gtk-main-quit is invoked while gtk-main is still running,  
gtk-main thus cleanly left.  Through gtk-quit callbacks open windows are  
closed, and gtk-main is called again.  This way no call stack piles up.   
In the normal case there should be no callstack building up this way.  You  
can check with (gtk-main-level).

Only errors in callbacks still lead to recursive re-entries.  Therefore I  
left the current termination code in place, though my changes should make  
this the exception rather than the normal case.

If someone figures out how to re-enter gtk-main after a callback signaled  
an error, I'd be very excited to learn.

- I added an additional error handler that catches lisp errors.  Thus, if  
a callback raises a lisp error, this is caught within the thread running  
gtk-main and a message box is displayed.  The user has the option to  
resignal the error (results in invoking the debugger) or to "recklessly  
continue" (which works fine as long as the callback has not caused  
inconsistent state).  This is helpful if (like me) you deal with user  
input and you prototype does not handle typos well

- cells-gtk has a major leak due to the nature it synchronized clos with  
gtk objects:  Upon creation, each clos object in cells-gtk is registered  
in *gtk-objects* along with the pointer to its gtk equivalent.  However,  
this entry is not removed when the gtk-instance is freed.  When  
subsequently new objects are created, gtk may well reuse this space,  
causing cells-gtk to signal an error like "new object has pointer x, but  
this is already known as such-and-such".  Curiously, it is usually only  
*gtk-objects* which holds a reference to an obsolete object.  For  
instance, running test-gtk allocates some 330 objects in *gtk-objects*,  
about 100 of which are not freed, when the app is closed.

The right way to solve this would be to hook into the on-delete-event  
signal of the widget object -- unfortunately it seems this is not called  
every time gtk deletes a widget.  I introduced some additional cleanup  
code in cells-gtk, thus covering about 70% of leftover widgets.

For the rest, I believe the right thing is to trust gtk when in doubt --  
thus I modified the gtk-objects registry to forget about the old object if  
gtk has given the pointer to a new one. (It used to throw an error and  
break gtk. I had it signal a warning in the beginning, but I came to  
believe that this cannot be the right way to deal with it; it would entail  
writing C-style memory management, carefully keeping track of objects and  
manually freeing them.  Ewww.)

This hasn't been a problem so far since it seems most cells-gtk apps just  
call cells-gtk-init before launching their app, thus wiping *gtk-objects*  
(and along with that all gtk state).  Not exactly dynamic ;).

For an example where the problem arises, consider the following slot def:

(caption-widgets :initform (c? (loop for caption in captions collecting  
(mk-label :text caption))))

Once you got to somehow free all the old labels everytime you create new  
ones, things get -- ehh -- ugly.

It looks like what we see here is the due penalty for using a C-library  
for doing our window management.  Maybe one day CLIM will be fast and  
sleek.

- For every new window, upon creation a gtk-quit callback is registered,  
closing that window once gtk-main-quit is called from within a callback.   
You force this to happen by setting terminate-on-close to t in a gtk-app  
window.

- I moved some initialization from start-app to an own function,  
init-gtk.  This one also takes care of multiple initialization.

- To prevent double initialization of glib's threading faacility via  
g-thread-init, there is a global variable in gtk-app.lisp.  This works  
fine, until you recompile this file.  So I figured why keep track of state  
when glib can do this for us?  It turns out there is a macro in glib for  
this purpose.  Since it's a macro (not a function), I added a respective  
function to gtk-adds.c.  A dependency on glib-2.0 is introduced, since  
threading is a glib feature, not gtk (this should be uncritical -- linux  
systems without glib are -- errr -- rare).  I added the corresponding  
stuff to Makefile.  Thus, when you compile cells-gtk with #+libcellsgtk,  
it will use this function.  Otherwise there is a flag variable in a  
closure around (init-gtk).

- On the same note, init-gtk checks whether the libs have beeen loaded by  
calling a function (gtk-main-level) from within a handler-case.  If this  
fails, we know they haven't been loaded.  I prefer to tap into exiting  
state information over duplicating it.

- The threading code is currently SBCL specific.  However, I singled out  
the needed primitives in four macro definitions, therefore it should be  
easy to add the respective calls for other implementations:


(defmacro thread-current ()
   #+sbcl `(identity sb-thread:*current-thread*)
   )

(defmacro thread-make (thread-fn name)
   (declare (ignorable name))
   #+sbcl `(sb-thread:make-thread ,thread-fn :name ,name)
   )

(defmacro thread-kill (thread)
   #+sbcl `(sb-thread:terminate-thread ,thread)
   )

(defmacro thread-yield ()
   #+sbcl `(sb-thread:thread-yield)
   )

- My changes will compile and run just fine without sbcl.  Everything  
should work normally, you just won't be able to use the threading  
functions like start-win and close-all-windows.