From sionescu at common-lisp.net Thu Jun 3 09:00:05 2010 From: sionescu at common-lisp.net (Stelian Ionescu) Date: Thu, 03 Jun 2010 05:00:05 -0400 Subject: [iolib-devel] New patches: 2-Jun-2010 Message-ID: commit 75fd5c4000b25ce394e0b62f0e227803eba47061 Author: Stelian Ionescu Date: Thu Jun 3 16:23:32 2010 +0800 Use IOLIB.BASE:CL-SOURCE-FILE as component-class for tests and examples examples/iolib.examples.asd | 4 ++++ tests/iolib-tests.asd | 4 ++++ 2 files changed, 8 insertions(+), 0 deletions(-) commit c6ed1551dc6e066db73cc49510ba952f27e1c548 Author: Stelian Ionescu Date: Thu Jun 3 16:22:54 2010 +0800 Add ISYS:SYSCALL-ERROR-P src/sockets/socket-methods.lisp | 8 ++++---- src/syscalls/conditions.lisp | 3 +++ src/syscalls/pkgdcl.lisp | 2 +- 3 files changed, 8 insertions(+), 5 deletions(-) commit 1473546c50449f4dd7a9876dcac1615f09b4b4f8 Author: Stelian Ionescu Date: Wed Jun 2 22:55:09 2010 +0800 Add git config stuff for special treatment of lisp and texinfo files .gitattributes | 1 + .gitconfig | 4 ++++ 2 files changed, 5 insertions(+), 0 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20100602.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git From archimag at gmail.com Fri Jun 4 09:56:17 2010 From: archimag at gmail.com (Andrey Moskvitin) Date: Fri, 4 Jun 2010 13:56:17 +0400 Subject: [iolib-devel] Library for run child processes Message-ID: Hi, I wrote a very simple library iolib.process, which allows you to run child processes and interact with them through the standard IO-streams. In contrast to the sb-ext:run-programm and similar tools offered by implementations, iolib.process not depend on the specific implementation, but only on iolib.syscalls and iolib.streams. iolib.process should work on all Unix-systems, tested on Linux with SBCL, Clozure CL and CLISP. Perhaps, after appropriate revision, it makes sense to include this library in the iolib. URL: http://github.com/archimag/iolib.process/ Moskvitin Andrey -------------- next part -------------- An HTML attachment was scrubbed... URL: From attila.lendvai at gmail.com Fri Jun 4 10:32:48 2010 From: attila.lendvai at gmail.com (Attila Lendvai) Date: Fri, 4 Jun 2010 12:32:48 +0200 Subject: [iolib-devel] Library for run child processes In-Reply-To: References: Message-ID: i didn't check what's inside, and i have zero posix knowledge around there anyways... but i very much miss a properly working exec without all those random surprises i've seen using sb-ext and trivial-shell... i hope Stelian will incorporate it into iolib. thanks, -- attila From david at lichteblau.com Fri Jun 4 10:37:59 2010 From: david at lichteblau.com (David Lichteblau) Date: Fri, 4 Jun 2010 12:37:59 +0200 Subject: [iolib-devel] Library for run child processes In-Reply-To: References: Message-ID: <20100604103759.GA17236@radon> Hi there, Quoting Andrey Moskvitin (archimag at gmail.com): > I wrote a very simple library iolib.process, which allows you to run child > processes and interact with them through the standard IO-streams. In > contrast > to the sb-ext:run-programm and similar tools offered by implementations, > iolib.process not depend on the specific implementation, but only on > iolib.syscalls > and iolib.streams. iolib.process should work on all Unix-systems, tested on > Linux with SBCL, Clozure CL and CLISP. Perhaps, after appropriate revision, > it makes sense to include this library in the iolib. > > URL: http://github.com/archimag/iolib.process/ having such a library sounds like a great idea, and I like your code in the sense that it looks somewhat similar architecturally to what I did when I needed something similar in Hemlock. Unfortunately, it would also run into the same problems as my code did: - On MacOS, SBCL doesn't survive a call to fork() if Lisp code in being run in the child process -- something about threading going wrong after the fork. The solution, unattractive as it may sound, is to write the code for the child process as a glue function written in C, which also implies doing the fork in C. - I'm a bit surprised that it works with CCL out of the box for you, because I recall having to disable GC or interrupts (or something like that) to by-pass a crash there. Perhaps writing the code in C isn't that bad an idea after all, because it also reduces this kind of portability issue. - When using the C code approach, some flexibility would get lost. In practise, user code often needs to set up the child process environment in ways that are hard to foresee for the library author, i.e. for FD redirection, tty and session handling, environment variables etc. (and attempts to implement a general API with lots of keyword arguments for those use cases does not lead to good API design, I think). What I would like to see is a little domain specific language that describes common syscalls and library functions (dup2, open, setenv, ...). It would then compile those calls into a byte array, and pass that to the C function. Following the fork, the C code would execute the bytecode. - As Stelian explained, there are certain issues with SIGCHLD that make this code unportable, because CLISP works very hard to keep iolib from getting its hands on the SIGCHLD handler. I think there are several approaches to this: a. Ignore the problem, declare CLISP unsupported. b. Solve the problem by clever SIGCHLD handler chaining. c. Write a separate "fork server", where the Lisp justs asks the server to spaws processes instead of doing that itself. (I believe Stelian wrote something like that, but I don't know where the code is.) d. Like c., but in particular run that "fork server" process as a child process of the Lisp. So the process hierarchy would be: | ^ | | socketpair for communication v v | | ... v v Forked process 1 Forked process 2 ... The advantage is that the helper process never dies, so it doesn't lead to SIGCHLD, and the SIGCHLDs for the other processes get handled in C. e. Distinguish between Lisps that need the hack described in d and those which don't. I.e., use method d on CLISP, but skip the helper process on SBCL, CCL and others. The disadvantage would be that code behaves differently depending on the Lisp used. Personally I would strongly prefer approaches a. or b. David From archimag at gmail.com Fri Jun 4 12:04:32 2010 From: archimag at gmail.com (Andrey Moskvitin) Date: Fri, 4 Jun 2010 16:04:32 +0400 Subject: [iolib-devel] Library for run child processes In-Reply-To: <20100604103759.GA17236@radon> References: <20100604103759.GA17236@radon> Message-ID: > On MacOS, SBCL doesn't survive a call to fork() if Lisp code in > being run in the child process -- something about threading going > wrong after the fork. Hmm, this behavior seems strange to me, I use "pure lisp" SBCL-daemon on Linux and do not have any problems with this, although the basic Lisp-code is executed after a fork. > The solution, unattractive as it may sound, is to write the code for > the child process as a glue function written in C, which also > implies doing the fork in C. I think that if Lisp-code after the fork will only have a few trivial foreign-calls, the behavior should not differ from the case use of C-code. However, I do not have MacOS, and I can not experiment with this. > I'm a bit surprised that it works with CCL out of the box for you, > because I recall having to disable GC or interrupts (or something > like that) to by-pass a crash there. See above. It seems to me that few trivial foreign-calls without memory allocation should not cause problems. > What I would like to see is a little domain specific language that > describes common syscalls and library functions (dup2, open, setenv, > ...). Now I use sb-ext:run-programma for rather simple tasks, such as to call sendmail. Use eDSL for this problem seems to me unnecessarily complicated. Yes, this approach could be interesting and very powerful, but not for the encountered my problems. IMHO, this solution on base eDSL should be completely independent library, because of its complexity increases the potential risks. But a simpler solution is also needed > a. Ignore the problem, declare CLISP unsupported. Yes, it pleases me most:) Andrey 2010/6/4 David Lichteblau > Hi there, > > Quoting Andrey Moskvitin (archimag at gmail.com): > > I wrote a very simple library iolib.process, which allows you to run > child > > processes and interact with them through the standard IO-streams. In > > contrast > > to the sb-ext:run-programm and similar tools offered by implementations, > > iolib.process not depend on the specific implementation, but only on > > iolib.syscalls > > and iolib.streams. iolib.process should work on all Unix-systems, tested > on > > Linux with SBCL, Clozure CL and CLISP. Perhaps, after appropriate > revision, > > it makes sense to include this library in the iolib. > > > > URL: http://github.com/archimag/iolib.process/ > > having such a library sounds like a great idea, and I like your code in > the sense that it looks somewhat similar architecturally to what I did > when I needed something similar in Hemlock. > > Unfortunately, it would also run into the same problems as my code did: > > - On MacOS, SBCL doesn't survive a call to fork() if Lisp code in > being run in the child process -- something about threading going > wrong after the fork. > > The solution, unattractive as it may sound, is to write the code for > the child process as a glue function written in C, which also > implies doing the fork in C. > > - I'm a bit surprised that it works with CCL out of the box for you, > because I recall having to disable GC or interrupts (or something > like that) to by-pass a crash there. > > Perhaps writing the code in C isn't that bad an idea after all, > because it also reduces this kind of portability issue. > > - When using the C code approach, some flexibility would get lost. In > practise, user code often needs to set up the child process > environment in ways that are hard to foresee for the library author, > i.e. for FD redirection, tty and session handling, environment > variables etc. (and attempts to implement a general API with lots of > keyword arguments for those use cases does not lead to good API > design, I think). > > What I would like to see is a little domain specific language that > describes common syscalls and library functions (dup2, open, setenv, > ...). It would then compile those calls into a byte array, and pass > that to the C function. Following the fork, the C code would > execute the bytecode. > > - As Stelian explained, there are certain issues with SIGCHLD that > make this code unportable, because CLISP works very hard to keep > iolib from getting its hands on the SIGCHLD handler. > > I think there are several approaches to this: > > a. Ignore the problem, declare CLISP unsupported. > > b. Solve the problem by clever SIGCHLD handler chaining. > > c. Write a separate "fork server", where the Lisp justs asks the > server to spaws processes instead of doing that itself. (I > believe Stelian wrote something like that, but I don't know > where the code is.) > > d. Like c., but in particular run that "fork server" process as a > child process of the Lisp. So the process hierarchy would be: > > > | ^ > | | socketpair for communication > v v > > | | ... > v v > Forked process 1 Forked process 2 ... > > The advantage is that the helper process never dies, so it > doesn't lead to SIGCHLD, and the SIGCHLDs for the other processes > get > handled in C. > > e. Distinguish between Lisps that need the hack described in d and > those which don't. I.e., use method d on CLISP, but skip the > helper process on SBCL, CCL and others. > > The disadvantage would be that code behaves differently > depending on the Lisp used. > > Personally I would strongly prefer approaches a. or b. > > > David > -------------- next part -------------- An HTML attachment was scrubbed... URL: From sionescu at cddr.org Sun Jun 6 19:34:57 2010 From: sionescu at cddr.org (Stelian Ionescu) Date: Mon, 07 Jun 2010 03:34:57 +0800 Subject: [iolib-devel] Library for run child processes In-Reply-To: <20100604103759.GA17236@radon> References: <20100604103759.GA17236@radon> Message-ID: <1275852897.2100.27.camel@blackhole.cddr.org> On Fri, 2010-06-04 at 12:37 +0200, David Lichteblau wrote: > Hi there, > > Quoting Andrey Moskvitin (archimag at gmail.com): > > I wrote a very simple library iolib.process, which allows you to run child > > processes and interact with them through the standard IO-streams. In > > contrast > > to the sb-ext:run-programm and similar tools offered by implementations, > > iolib.process not depend on the specific implementation, but only on > > iolib.syscalls > > and iolib.streams. iolib.process should work on all Unix-systems, tested on > > Linux with SBCL, Clozure CL and CLISP. Perhaps, after appropriate revision, > > it makes sense to include this library in the iolib. > > > > URL: http://github.com/archimag/iolib.process/ > > having such a library sounds like a great idea, and I like your code in > the sense that it looks somewhat similar architecturally to what I did > when I needed something similar in Hemlock. > > Unfortunately, it would also run into the same problems as my code did: > > - On MacOS, SBCL doesn't survive a call to fork() if Lisp code in > being run in the child process -- something about threading going > wrong after the fork. > > The solution, unattractive as it may sound, is to write the code for > the child process as a glue function written in C, which also > implies doing the fork in C. > > - I'm a bit surprised that it works with CCL out of the box for you, > because I recall having to disable GC or interrupts (or something > like that) to by-pass a crash there. > > Perhaps writing the code in C isn't that bad an idea after all, > because it also reduces this kind of portability issue. > > - When using the C code approach, some flexibility would get lost. In > practise, user code often needs to set up the child process > environment in ways that are hard to foresee for the library author, > i.e. for FD redirection, tty and session handling, environment > variables etc. (and attempts to implement a general API with lots of > keyword arguments for those use cases does not lead to good API > design, I think). > > What I would like to see is a little domain specific language that > describes common syscalls and library functions (dup2, open, setenv, > ...). It would then compile those calls into a byte array, and pass > that to the C function. Following the fork, the C code would > execute the bytecode. > > - As Stelian explained, there are certain issues with SIGCHLD that > make this code unportable, because CLISP works very hard to keep > iolib from getting its hands on the SIGCHLD handler. > > I think there are several approaches to this: > > a. Ignore the problem, declare CLISP unsupported. > > b. Solve the problem by clever SIGCHLD handler chaining. > [snip] > Personally I would strongly prefer approaches a. or b. Thanks for the code, Andrey. I added to iolib.os a different implementation that calls posix_spawn(3) because, as David said, it's unsafe to do the fork()-ing from Lisp code because we don't know how that might interact with the garbage collector. The implementation is currently not very complete, but for the moment you can (create-process "ls" (list "/tmp")) or (run-program "ls" (list "/tmp")) and it works As for the interactions with the implementation's run-program, I'm inclined to do something like this on every implementation: (defvar *host-run-program-replaced* nil) #+sbcl (when (not *host-run-program-replaced*) (let ((old-fn (fdefinition 'sb-ext:run-program))) (defun sb-ext:run-program (&rest args) (cerror "Continue using RUN-PROGRAM, but be warned that it will interfere with IOLIB.OS" "You're using RUN-PROGRAM while IOLIB.OS is loaded !") (apply old-fn args))) (setf *host-run-program-replaced* t)) -- Stelian Ionescu a.k.a. fe[nl]ix Quidquid latine dictum sit, altum videtur. http://common-lisp.net/project/iolib -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 198 bytes Desc: This is a digitally signed message part URL: From sionescu at common-lisp.net Mon Jun 7 09:00:05 2010 From: sionescu at common-lisp.net (Stelian Ionescu) Date: Mon, 07 Jun 2010 05:00:05 -0400 Subject: [iolib-devel] New patches: 6-Jun-2010 Message-ID: commit 0e76862590dc75ff6856c5bd7873ac0d84082ae1 Author: Stelian Ionescu Date: Mon Jun 7 03:05:18 2010 +0800 Export process-related symbols src/os/pkgdcl.lisp | 11 +++++++++++ 1 files changed, 11 insertions(+), 0 deletions(-) commit a0a04f588a2aa023c2c666a03071fb0dd954d7d2 Author: Stelian Ionescu Date: Mon Jun 7 02:59:39 2010 +0800 Add CREATE-PROCESS src/iolib.os.asd | 10 ++- src/os/create-process-unix.lisp | 128 +++++++++++++++++++++++++++++++++++++++ src/os/ffi-functions-unix.lisp | 119 ++++++++++++++++++++++++++++++++++++ src/os/ffi-types-unix.lisp | 25 ++++++++ src/os/os-unix.lisp | 32 ++++++++++ src/os/pkgdcl.lisp | 2 + 6 files changed, 313 insertions(+), 3 deletions(-) commit 27781f6590c8fb6e743e64a22d9f3bdb48c203e5 Author: Stelian Ionescu Date: Sun Jun 6 20:35:30 2010 +0800 Fix .asd files for ASDF2 examples/iolib.examples.asd | 49 +++++++++++++++++++++-------------------- src/iolib.asd | 2 +- src/iolib.multiplex.asd | 2 +- src/iolib.os.asd | 6 ++-- src/iolib.pathnames.asd | 6 ++-- src/iolib.sockets.asd | 26 +++++++++++----------- src/iolib.streams.asd | 2 +- src/iolib.syscalls.asd | 6 ++-- src/iolib.trivial-sockets.asd | 2 +- src/iolib.zstreams.asd | 2 +- tests/iolib-tests.asd | 16 ++++++------ 11 files changed, 60 insertions(+), 59 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20100606.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git From archimag at gmail.com Tue Jun 8 08:12:10 2010 From: archimag at gmail.com (Andrey Moskvitin) Date: Tue, 8 Jun 2010 12:12:10 +0400 Subject: [iolib-devel] Library for run child processes In-Reply-To: <1275852897.2100.27.camel@blackhole.cddr.org> References: <20100604103759.GA17236@radon> <1275852897.2100.27.camel@blackhole.cddr.org> Message-ID: Stelian, > I added to iolib.os a different > implementation that calls posix_spawn(3) Great, I agree that posix_spawn is a better solution than fork/exec. > The implementation is currently not very complete I rewrite your code, so that it now passes my tests. I tested it on Linux with SBCL, Clozure CL and CLISP. My code in fork iolib: http://gitorious.org/~archimag/iolib/archimag-iolib/blobs/master/src/os/create-process-unix.lisp My tests: http://gitorious.org/iolib-test/iolib-test/blobs/master/process.lisp Andrey -------------- next part -------------- A non-text attachment was scrubbed... Name: iolib-create-process.patch Type: text/x-patch Size: 10155 bytes Desc: not available URL: From sionescu at common-lisp.net Wed Jun 9 09:00:05 2010 From: sionescu at common-lisp.net (Stelian Ionescu) Date: Wed, 09 Jun 2010 05:00:05 -0400 Subject: [iolib-devel] New patches: 8-Jun-2010 Message-ID: commit 577af3315b8333c2f2358edc873073050674214f Author: Stelian Ionescu Date: Tue Jun 8 18:59:49 2010 +0800 Only grovel O_CLOEXEC on Linux src/syscalls/ffi-types-unix.lisp | 3 +-- src/syscalls/pkgdcl.lisp | 2 +- 2 files changed, 2 insertions(+), 3 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20100608.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git From sionescu at common-lisp.net Wed Jun 23 09:00:05 2010 From: sionescu at common-lisp.net (Stelian Ionescu) Date: Wed, 23 Jun 2010 05:00:05 -0400 Subject: [iolib-devel] New patches: 22-Jun-2010 Message-ID: commit 35772a37d1ede31e60c49728f91435e5e0e7f502 Author: Stelian Ionescu Date: Wed Jun 23 14:57:57 2010 +0800 STREAM-FORCE-OUTPUT now tries to flush the output buffer without blocking src/streams/gray/gray-stream-methods.lisp | 10 +++++++++- src/streams/gray/io-helpers.lisp | 12 ++++++++---- 2 files changed, 17 insertions(+), 5 deletions(-) commit fd6e5284adbf4898e811d485539b0d31d71cb856 Author: Francois-Rene Rideau Date: Wed Jun 23 02:20:45 2010 +0800 Wrap WIFSIGNALED src/syscalls/ffi-wrappers-unix.lisp | 3 +++ src/syscalls/pkgdcl.lisp | 1 + 2 files changed, 4 insertions(+), 0 deletions(-) commit 07ecb0e24d8fd7dfeab0f0bc19edc523e33d8502 Author: Francois-Rene Rideau Date: Wed Jun 23 02:20:11 2010 +0800 Add sysconf(3) src/syscalls/ffi-functions-unix.lisp | 8 ++++++++ src/syscalls/ffi-types-unix.lisp | 24 ++++++++++++++++++++++++ src/syscalls/pkgdcl.lisp | 8 ++++++++ 3 files changed, 40 insertions(+), 0 deletions(-) commit 71c7d0852ce684978b5d5417b48ae0ece9b29ab8 Author: Stelian Ionescu Date: Wed Jun 23 02:11:43 2010 +0800 Fix typo src/syscalls/ffi-functions-unix.lisp | 2 +- src/syscalls/ffi-types-unix.lisp | 1 + 2 files changed, 2 insertions(+), 1 deletions(-) commit 69fed8e73c28d2551ada798b34445252c3aa149c Author: Stelian Ionescu Date: Sun Jun 13 02:19:51 2010 +0800 Make all stream FDs non-blocking, not just sockets src/sockets/socket-methods.lisp | 3 +-- src/streams/gray/fd-mixin.lisp | 6 ++++++ 2 files changed, 7 insertions(+), 2 deletions(-) commit 4064fa95460acb50212ac9592d21caa6d124c7c3 Author: Stelian Ionescu Date: Sun Jun 13 02:19:23 2010 +0800 Add PRINT-OBJECT for class DUAL-CHANNEL-GRAY-STREAM src/streams/gray/gray-stream-methods.lisp | 16 ++++++++++++++++ 1 files changed, 16 insertions(+), 0 deletions(-) commit 6691415feba7a0fcd00098674e6a54a07e26aa0c Author: Stelian Ionescu Date: Thu Jun 10 00:36:13 2010 +0800 Remove class DUAL-CHANNEL-SINGLE-FD-MIXIN src/os/create-process-unix.lisp | 9 ++---- src/sockets/base-sockets.lisp | 2 +- src/streams/gray/classes.lisp | 38 +--------------------------- src/streams/gray/fd-mixin.lisp | 24 ++---------------- src/streams/gray/gray-stream-methods.lisp | 8 +++--- src/streams/gray/io-helpers.lisp | 6 ++-- src/streams/gray/pkgdcl.lisp | 6 ---- tests/streams.lisp | 5 +-- 8 files changed, 18 insertions(+), 80 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20100622.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git From sionescu at common-lisp.net Thu Jun 24 09:00:08 2010 From: sionescu at common-lisp.net (Stelian Ionescu) Date: Thu, 24 Jun 2010 05:00:08 -0400 Subject: [iolib-devel] New patches: 23-Jun-2010 Message-ID: commit a02459ea002b564328a6900a266bc726d057640f Author: Stelian Ionescu Date: Wed Jun 23 16:48:56 2010 +0800 Fix socket initialization src/sockets/socket-methods.lisp | 14 ++++++++------ src/streams/gray/fd-mixin.lisp | 7 +++---- 2 files changed, 11 insertions(+), 10 deletions(-) An updated tarball of IOLib's source can be downloaded here: http://common-lisp.net/project/iolib/files/snapshots/iolib-20100623.tar.gz Gitweb URL: http://repo.or.cz/w/iolib.git