[iolib-devel] Library for run child processes

[Stelian Ionescu]

On Fri, 2010-06-04 at 12:37 +0200, David Lichteblau wrote:
> Hi there,
> 
> Quoting Andrey Moskvitin (archimag at gmail.com):
> > I wrote a very simple library iolib.process, which allows you to run child
> > processes and interact with them through the standard IO-streams. In
> > contrast
> > to the sb-ext:run-programm and similar tools offered by implementations,
> > iolib.process not depend on the specific implementation, but only on
> > iolib.syscalls
> > and iolib.streams. iolib.process should work on all Unix-systems, tested on
> > Linux with SBCL, Clozure CL and CLISP. Perhaps, after appropriate revision,
> > it makes sense to include this library in the iolib.
> > 
> > URL: http://github.com/archimag/iolib.process/
> 
> having such a library sounds like a great idea, and I like your code in
> the sense that it looks somewhat similar architecturally to what I did
> when I needed something similar in Hemlock.
> 
> Unfortunately, it would also run into the same problems as my code did:
> 
>   - On MacOS, SBCL doesn't survive a call to fork() if Lisp code in
>     being run in the child process -- something about threading going
>     wrong after the fork.
> 
>     The solution, unattractive as it may sound, is to write the code for
>     the child process as a glue function written in C, which also
>     implies doing the fork in C.
> 
>   - I'm a bit surprised that it works with CCL out of the box for you,
>     because I recall having to disable GC or interrupts (or something
>     like that) to by-pass a crash there.
> 
>     Perhaps writing the code in C isn't that bad an idea after all,
>     because it also reduces this kind of portability issue.
> 
>   - When using the C code approach, some flexibility would get lost.  In
>     practise, user code often needs to set up the child process
>     environment in ways that are hard to foresee for the library author,
>     i.e. for FD redirection, tty and session handling, environment
>     variables etc.  (and attempts to implement a general API with lots of
>     keyword arguments for those use cases does not lead to good API
>     design, I think).
> 
>     What I would like to see is a little domain specific language that
>     describes common syscalls and library functions (dup2, open, setenv, 
>     ...).  It would then compile those calls into a byte array, and pass
>     that to the C function.  Following the fork, the C code would
>     execute the bytecode.
> 
>   - As Stelian explained, there are certain issues with SIGCHLD that
>     make this code unportable, because CLISP works very hard to keep
>     iolib from getting its hands on the SIGCHLD handler.
> 
>     I think there are several approaches to this:
> 
>      a. Ignore the problem, declare CLISP unsupported.
> 
>      b. Solve the problem by clever SIGCHLD handler chaining.
> [snip]

> Personally I would strongly prefer approaches a. or b.

Thanks for the code, Andrey. I added to iolib.os a different
implementation that calls posix_spawn(3) because, as David said, it's
unsafe to do the fork()-ing from Lisp code because we don't know how
that might interact with the garbage collector.

The implementation is currently not very complete, but for the moment
you can (create-process "ls" (list "/tmp")) or
(run-program "ls" (list "/tmp")) and it works

As for the interactions with the implementation's run-program, I'm
inclined to do something like this on every implementation:

(defvar *host-run-program-replaced* nil)

#+sbcl
(when (not *host-run-program-replaced*)
  (let ((old-fn (fdefinition 'sb-ext:run-program)))
    (defun sb-ext:run-program (&rest args)
      (cerror "Continue using RUN-PROGRAM, but be warned that it will interfere with IOLIB.OS"
              "You're using RUN-PROGRAM while IOLIB.OS is loaded !")
      (apply old-fn args)))
  (setf *host-run-program-replaced* t))

-- 
Stelian Ionescu a.k.a. fe[nl]ix
Quidquid latine dictum sit, altum videtur.
http://common-lisp.net/project/iolib
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <https://mailman.common-lisp.net/pipermail/iolib-devel/attachments/20100607/6bdfc622/attachment.sig>