From smustudent1 at yahoo.com Fri Jul 1 14:56:22 2005 From: smustudent1 at yahoo.com (C Y) Date: Fri, 1 Jul 2005 07:56:22 -0700 (PDT) Subject: [fetter-devel] Question Message-ID: <20050701145622.49395.qmail@web31711.mail.mud.yahoo.com> I'm not really technically qualified to discuss FFIs, but I do have an interst in them as a possible future aid to the Maxima project. I have a question as to the "scale" of FFIs, and it looks like this is the place to ask :-). AFAICT, there seem to be two levels, or maybe philosophies, to Lisp FFI interactions. The first is low level, doing the minimal mapping needed to be able to interact with foreign library constructs in Lisp. The second is much higher level (and maybe more ambitious?) - to make interacting with the library in question a "lisp-like" experience - i.e. the interaction between programmer and library takes place at a level where the details of the foreign nature of the library can be safely ignored. Have I understood this correctly? If so, my question is whether these goals are complementary or mutually exclusive. I have heard good things about both Allegro's and Clisp's FFIs, but IIRC those are both "high level." Is there some reason the "high level" activity can't be built on the lower level work, with both accessable? I apologize if this is obvious or the wrong place to ask what must be a rather basic question. Cheers, CY __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From ktilton at nyc.rr.com Fri Jul 1 15:17:02 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Fri, 01 Jul 2005 11:17:02 -0400 Subject: [fetter-devel] Question In-Reply-To: <20050701145622.49395.qmail@web31711.mail.mud.yahoo.com> References: <20050701145622.49395.qmail@web31711.mail.mud.yahoo.com> Message-ID: <42C55E6E.4080503@nyc.rr.com> C Y wrote: >I'm not really technically qualified to discuss FFIs, but I do have an >interst in them as a possible future aid to the Maxima project. I have >a question as to the "scale" of FFIs, and it looks like this is the >place to ask :-). > >AFAICT, there seem to be two levels, or maybe philosophies, to Lisp FFI >interactions. The first is low level, doing the minimal mapping needed >to be able to interact with foreign library constructs in Lisp. The >second is much higher level (and maybe more ambitious?) - to make >interacting with the library in question a "lisp-like" experience - >i.e. the interaction between programmer and library takes place at a >level where the details of the foreign nature of the library can be >safely ignored. Have I understood this correctly? > >If so, my question is whether these goals are complementary or mutually >exclusive. I have heard good things about both Allegro's and Clisp's >FFIs, but IIRC those are both "high level." Is there some reason the >"high level" activity can't be built on the lower level work, with both >accessable? > No, that is a sensible approach. You can even have three levels, with a new higher level doing even lispier things, such as taking output parameters to a C function (I pass a pointer, the function writes to it) and returning them along with the function result in a values form. Check out the CFFI project, also on c-l.net. James is going for low-level FFI of the type I think you mean. Hello-C, a sister project of Fetter, is looking closely at that as the implementation mechanism for a truly portable CL FFI. -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From luismbo at gmail.com Sat Jul 2 03:01:58 2005 From: luismbo at gmail.com (Luis Oliveira) Date: Sat, 2 Jul 2005 04:01:58 +0100 Subject: [fetter-devel] GCCXML -> UFFI tool Message-ID: <6b5fa9507b197e85dfa8681ef28cbb2b@gmail.com> Rayiner, In case you don't read Planet Lisp, here's a recent post that mentions a tool to generate UFFI bindings from GCCXML output: http://www.cyrusharmon.org/cl/blog/display/24 -- Lu?s Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www2.iro.umontreal.ca/~pinard/po/registry.cgi?team=pt From ktilton at nyc.rr.com Sat Jul 2 05:19:23 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Sat, 02 Jul 2005 01:19:23 -0400 Subject: [fetter-devel] GCCXML -> UFFI tool In-Reply-To: <6b5fa9507b197e85dfa8681ef28cbb2b@gmail.com> References: <6b5fa9507b197e85dfa8681ef28cbb2b@gmail.com> Message-ID: <42C623DB.8040301@nyc.rr.com> Luis Oliveira wrote: > Rayiner, > > In case you don't read Planet Lisp, here's a recent post that mentions > a tool to generate UFFI bindings from GCCXML output: > > http://www.cyrusharmon.org/cl/blog/display/24 > And we have not even made formal announcements. When we do, looks like we will discover even more related work. My advice? Take a look, contact the developer to see how active they are, and figure out as efficiently as possible how much added value another project brings, ask the community for its input (cll, PlanetLisp, Lisp #IRC, etc). Usually I can write something from scratch as fast as I can figure out some one else's system, which probably will need modification or extension, but now I am working on someone else's code, which may not be as good and which I definitely need to learn before I even know if it is any good. We could spend the whole summer looking at OPC (Other People's Code). It is great if we discover a gem of open source, but let's not spend the summer chasing OPC in the hope of somehow saving time in spite of the search and learning curve and probably repair/extension curve of OPC. -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From jamesjb at jamesjb.com Sat Jul 2 20:29:46 2005 From: jamesjb at jamesjb.com (James Bielman) Date: Sat, 02 Jul 2005 13:29:46 -0700 Subject: [fetter-devel] ECL In-Reply-To: <42C3A445.20008@mpq.mpg.de> (Juan Jose Garcia Ripoll's message of "Thu, 30 Jun 2005 09:50:29 +0200") References: <42C2DBCF.1050705@mpq.mpg.de> <085e506782f4543c73cdcc98de3d66d9@gmail.com> <42C3A445.20008@mpq.mpg.de> Message-ID: Juan Jose Garcia Ripoll writes: > I have seen you have a single primitive, "%foreing-function-call", > which can call a pointer with any number of types. Even though this > may be implementable even in C (with a type cast), I do not very > much like the idea. Personally I prefer wrappers. The reason is > that, unlike other lisps, ECL can link its code against that of the > C libraries. Currently, there is redundancy because we use not only > the type declarations provided by the user in the FFI, but also > those of the C headers of the code we are to link against. This > means the C compiler can double-check the declarations made by the > lisper and ensure everything will go right. I've expected there to be some controversy about FOREIGN-FUNCALL. The main reason I like this operator is because it becomes very easy to call variable-argument functions, and one of my long-term goals with CFFI is to write a portable Objective-C bridge. Using FOREIGN-FUNCALL you can define a primitive for calling ObjC methods almost trivially: (defmacro send (object method-name &rest args) "Call an Objective-C method METHOD-NAME on OBJECT. If OBJECT is a string, a method on the class of that name will be invoked." (when (stringp object) (setf object `(find-objc-class ,object))) `(foreign-funcall "objc_msgSend" :pointer ,object :pointer (selector ,method-name) , at args)) ? (send (send "NSAutoreleasePool" "alloc" :pointer) "init" :pointer) # It is also easy to experiment with calling foreign functions from the REPL when you don't have to declare them first: ? (foreign-funcall "sqrtf" :float 16.0 :float) 4.0 Of course, CFFI does provide the DEFCFUN macro for generating Lisp wrappers to call foreign functions, but these wrappers are implemented portably in terms of FOREIGN-FUNCALL, and are able to do automatic argument translation (Lisp to foreign strings, and vice versa). (defcfun "puts" :void (s string)) ==> (defun puts (s) (foreign-funcall "puts" string s :void)) ;; For fun, here's what this FOREIGN-FUNCALL macroexpands into. This ;; really should stack-allocate the foreign string but it doesn't yet. (let ((#:g8 (foreign-string-alloc s))) (unwind-protect (let ((#:g9 (%foreign-funcall "puts" :pointer #:g8 :void))) #:g9) (foreign-string-free #:g8))) The user can also define his own type translators for arguments and return values. Here is how the STRING type translator looks (this interface will probably change soon, it doesn't need to be this complicated): ;; Define STRING as a typedef for :POINTER. (defctype string :pointer) (define-type-translator string :in (arg result-var) "Type translator for string input arguments." (values `(foreign-string-alloc ,arg) ; result `((foreign-string-free ,result-var)))) ; unwinds (define-type-translator string :result (arg result-var) "Type translator for string return values." (declare (ignore result-var)) (values `(foreign-string-to-lisp ,arg) ; result nil)) ; unwinds James From jamesjb at jamesjb.com Sat Jul 2 20:48:51 2005 From: jamesjb at jamesjb.com (James Bielman) Date: Sat, 02 Jul 2005 13:48:51 -0700 Subject: [fetter-devel] Re: CFFI progress In-Reply-To: <42C4126D.1050504@nyc.rr.com> (Kenny Tilton's message of "Thu, 30 Jun 2005 11:40:29 -0400") References: <55e3b745aa12e271ddcb05424a812c84@gmail.com> <42C36DC6.1080203@nyc.rr.com> <42C4126D.1050504@nyc.rr.com> Message-ID: Kenny Tilton writes: > Well, no, the goal is a truly universal FFI, in both functionality > (adding callbacks) and Lisps supported. We /thought/ that would mean > extending UFFI, but CFFI has us wondering if a better approach is > possible. And let's not forget the possibility of a hybrid > approach. I know UFFI on CMUCL has a history of performance > problems. Maybe we go the CFFI route with CMUCL and let other Lisps' > FFIs handle things themselves? While this may be possible, it seems to me like it may be a lot of extra work. IMHO if you're going to go to the trouble of implementing your own foreign type system, you may as well make it portable. If you really want a hybrid approach, it may be easier to try to use the alien interface to obtain information at macroexpansion time but still use it for defining foreign types (handwavy but I dont know the details, most likely you'd need to get some stuff exported). > I see two concerns: > > - will every Lisp support CFFI with the necessary low-level ops > > - are the low-level ops exported / stable / reliable / supported? > > - will CFFI sometimes be less effective than the implementation FFI? > For example, Franz indicates their native sockets package works > better with their MP code, so in MP situations would run better than > a generic solution using winsock or native BSD sockets via the FFI. While I'm not so ambitious as to try to write some sort of CLRFI and get every CL vendor to implement it, I am trying to take a slightly longer-term view, and at least convince each vendor to make a core set of functionality available and documented/exported/etc. For example, the reason 'cffi-allegro.lisp' doesn't exist yet is because I want to get Franz's input on how best to implement the CFFI-SYS specification for their Lisp, as they do not export some of the pieces I need. (Unfortunately I have essentially no time to work on any of this for most of July, but it will get done eventually). > That was just a long argument against worrying about backward > compatibility. Ah yes, who cares about that?! :) >> In CFFI, I've pretty much thrown out the UFFI interface and written >> my own. In particular, everything that can conceivably be a >> function is a function, in contrast to UFFI which consists almost >> entirely of macros (many of which can be confusing WRT what gets >> evaluated on which Lisp). >> > Yep. I think KMR realizes this and would cure in UFFI2. It's hard though, because the host Lisp FFIs can vary wildy in how they evaluate their arguments. For example, to use OpenMCL's high level FFI, almost everything has to be constant at macroexpansion time. UFFI even has to do some gross things like intern keyword symbols at macroexpansion time for building structure references. I don't see that there's any way around these sorts of problems as long as you are trying to paper over each Lisp's high-level interface. >> CFFI-UFFI-COMPAT has quite a bit of value to me because I can build >> existing libraries like CLSQL, get a performance boost on SBCL, and >> not have to change the code (and indeed I've successfully built the >> CLSQL-SQLite3 backend using CFFI-UFFI-COMPAT). >> > Yep. Was that no changes at all, or "just a few" tweaks? Up above I > was saying that, if and only if we had to break backward compatibility > with UFFI, that that would not be the end of the world if we kept the > breakage to a minimum. No changes at all, but I did read through the code first to make sure it didn't have any embedded arrays or nested structures, since the compatibility layer doesn't handle these right yet. The compatibility layer was basically a for-fun afterthought I had, and if I can't make it work for the bulk of the UFFI-ized libraries out there, I'll probably scrap it. James From rayiner at gmail.com Sat Jul 2 21:00:07 2005 From: rayiner at gmail.com (Rayiner Hashem) Date: Sat, 2 Jul 2005 17:00:07 -0400 Subject: [fetter-devel] GCCXML -> UFFI tool In-Reply-To: <42C623DB.8040301@nyc.rr.com> References: <6b5fa9507b197e85dfa8681ef28cbb2b@gmail.com> <42C623DB.8040301@nyc.rr.com> Message-ID: > Take a look, contact the developer to see how active they are, and > figure out as efficiently as possible how much added value another > project brings, ask the community for its input (cll, PlanetLisp, Lisp > #IRC, etc). I've taken a look at the code (I'll get in contact with the developer later this weekend), and it doesn't seem _that_ far along. It's actually a very straightforward translation of GCC-XML to UFFI, only handling C, along the lines of the prototype I built last year. I don't think it would save us a massive amount of time to start with it as the base instead of just building our own. > Usually I can write something from scratch as fast as I can figure out > some one else's system, which probably will need modification or > extension, but now I am working on someone else's code, which may not be > as good and which I definitely need to learn before I even know if it is > any good. We could spend the whole summer looking at OPC (Other People's > Code). > > It is great if we discover a gem of open source, but let's not spend the > summer chasing OPC in the hope of somehow saving time in spite of the > search and learning curve and probably repair/extension curve of OPC. I agree. The idea of using GCC-XML isn't new (I've seen it before on c.l.l), though this is the first implementation of it I've seen. So I'll take a look at what's available, however, my schedule for fetter is tight and my priority is getting done the things I promised to do in my proposal within the time available. From ktilton at nyc.rr.com Sun Jul 3 02:43:54 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Sat, 02 Jul 2005 22:43:54 -0400 Subject: [fetter-devel] Re: CFFI progress In-Reply-To: References: <55e3b745aa12e271ddcb05424a812c84@gmail.com> <42C36DC6.1080203@nyc.rr.com> <42C4126D.1050504@nyc.rr.com> Message-ID: <42C750EA.5080500@nyc.rr.com> James Bielman wrote: >Kenny Tilton writes: > > >>I see two concerns: >> >>- will every Lisp support CFFI with the necessary low-level ops >> >>- are the low-level ops exported / stable / reliable / supported? >> >>- will CFFI sometimes be less effective than the implementation FFI? >> For example, Franz indicates their native sockets package works >> better with their MP code, so in MP situations would run better than >> a generic solution using winsock or native BSD sockets via the FFI. >> >> > >While I'm not so ambitious as to try to write some sort of CLRFI and >get every CL vendor to implement it, I am trying to take a slightly >longer-term view, and at least convince each vendor to make a core set >of functionality available and documented/exported/etc. > >For example, the reason 'cffi-allegro.lisp' doesn't exist yet is >because I want to get Franz's input on how best to implement the >CFFI-SYS specification for their Lisp, as they do not export some of >the pieces I need. > >(Unfortunately I have essentially no time to work on any of this for >most of July, but it will get done eventually). > >>>In CFFI, I've pretty much thrown out the UFFI interface and written >>>my own. In particular, everything that can conceivably be a >>>function is a function, in contrast to UFFI which consists almost >>>entirely of macros (many of which can be confusing WRT what gets >>>evaluated on which Lisp). >>> >>> >>> >>Yep. I think KMR realizes this and would cure in UFFI2. >> >> > >It's hard though, because the host Lisp FFIs can vary wildy in how >they evaluate their arguments. For example, to use OpenMCL's high >level FFI, almost everything has to be constant at macroexpansion >time. > OK, and there were times I wanted UFFI macro args evaluated and they were not, but not many. > UFFI even has to do some gross things like intern keyword >symbols at macroexpansion time for building structure references. > >I don't see that there's any way around these sorts of problems as >long as you are trying to paper over each Lisp's high-level >interface. > OK, but is this a fair summary?: Option A: The CFFI approach bypassing native Lisp FFIs. Works only on some Lisps. (Which?) The other Lisps have to be persuaded to expose and export some internals to support CFFI. Option B: Wrap native Lisp FFIs, accepting certain limitations such as arguments not being evaluated. (Exactly which arguments to which FFI entry points?) Persuade some Lisps to change their native FFIs to evaluate certain arguments. With Option A, at summer's end only some Lisps are supported. Optimization of Native FFIs is lost. With Option B, i think by summer's end we have a portable, truly universal FFI, with the limitation that on some Lisps some arguments are not evaluated. Optimization of native FFIs is retained. >The compatibility layer was basically a for-fun afterthought I had, >and if I can't make it work for the bulk of the UFFI-ized libraries >out there, I'll probably scrap it. > Understood, and agreed. KMR himself agrees UFFI has run its course. Existing UFFI bindings will take no more than a day to transform to something newer, so let's not handcuff the next generation. -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From ktilton at nyc.rr.com Sun Jul 3 04:09:28 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Sun, 03 Jul 2005 00:09:28 -0400 Subject: [fetter-devel] Re: CFFI progress In-Reply-To: <42C750EA.5080500@nyc.rr.com> References: <55e3b745aa12e271ddcb05424a812c84@gmail.com> <42C36DC6.1080203@nyc.rr.com> <42C4126D.1050504@nyc.rr.com> <42C750EA.5080500@nyc.rr.com> Message-ID: <42C764F8.6060901@nyc.rr.com> Kenny Tilton wrote: > > > James Bielman wrote: > >> It's hard though, because the host Lisp FFIs can vary wildy in how >> they evaluate their arguments. For example, to use OpenMCL's high >> level FFI, almost everything has to be constant at macroexpansion >> time. > > OK, and there were times I wanted UFFI macro args evaluated and they > were not, but not many. > >> UFFI even has to do some gross things like intern keyword >> symbols at macroexpansion time for building structure references. >> >> I don't see that there's any way around these sorts of problems as >> long as you are trying to paper over each Lisp's high-level >> interface. >> > OK, but is this a fair summary?: > > Option A: The CFFI approach bypassing native Lisp FFIs. Works only on > some Lisps. (Which?) The other Lisps have to be persuaded to expose > and export some internals to support CFFI. > > Option B: Wrap native Lisp FFIs, accepting certain limitations such as > arguments not being evaluated. (Exactly which arguments to which FFI > entry points?) Persuade some Lisps to change their native FFIs to > evaluate certain arguments. > > With Option A, at summer's end only some Lisps are supported. > Optimization of Native FFIs is lost. > > With Option B, i think by summer's end we have a portable, truly > universal FFI, with the limitation that on some Lisps some arguments > are not evaluated. Optimization of native FFIs is retained. Btw, just to be clear, I love clean breaks with the past, which is why I kinda like the idea of a hybrid solution in which Lisps not ready for CFFI are supported via their native FFIs, the problem obviously being finding an api they all will support (perhaps, again, losing some power here and there where native FFI macros do not evaluate their arguments. And if finding that common FFI becomes tedious, than it may well be wise to limit the summer project to doing a brilliant job extending CFFI to the utmost. What I would like to see is a list of which Lisps: a. fully support CFFI with exported, supported hooks b. support CFFI with unsupported hooks c. can support the CFFI API without compromise thx to native FFI d. can support the CFFI API with exactly what compromises using native FFI That landscape will inform the debate nicely, methinks. -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From luismbo at gmail.com Sun Jul 3 12:14:59 2005 From: luismbo at gmail.com (Luis Oliveira) Date: Sun, 3 Jul 2005 13:14:59 +0100 Subject: [fetter-devel] Re: CFFI progress In-Reply-To: <42C764F8.6060901@nyc.rr.com> References: <55e3b745aa12e271ddcb05424a812c84@gmail.com> <42C36DC6.1080203@nyc.rr.com> <42C4126D.1050504@nyc.rr.com> <42C750EA.5080500@nyc.rr.com> <42C764F8.6060901@nyc.rr.com> Message-ID: On 3/jul/2005, at 05:09, Kenny Tilton wrote: > a. fully support CFFI with exported, supported hooks > b. support CFFI with unsupported hooks I think the best way to figure this out is to port the existing features to other lisps. (I'm sure you want Allegro support ASAP :). I started by adding foreign vars (should have a patch for sbcl/clisp/cmucl/openmcl with a basic interface sometime today) and I was thinking about going for callbacks next. > c. can support the CFFI API without compromise thx to native FFI > d. can support the CFFI API with exactly what compromises using native > FFI Again, I suppose the best way to figure this out is to port cffi to other lisps? -- Lu?s Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www2.iro.umontreal.ca/~pinard/po/registry.cgi?team=pt From luismbo at gmail.com Mon Jul 4 03:44:50 2005 From: luismbo at gmail.com (Luis Oliveira) Date: Mon, 4 Jul 2005 04:44:50 +0100 Subject: [fetter-devel] Quick CFFI update Message-ID: <63d454e0eb25cc299a16c89b2514b861@gmail.com> [Rayiner/Kenny these mails should be sent to cffi-devel only right? If you agree, I'll be sending future mails about CFFI to cffi-devel only.] Hello, I pushed my preliminary support for foreign globals to my darcs CFFI tree: http://common-lisp.net/project/cffi/darcs/cffi-luis It works with sbcl/openmcl/clisp/cmucl (actually I haven't tested with clisp yet) but requires a patch to clisp[1] and to sbcl[2] too. I'm interested in your comments, and I had few questions in my notes but I'll leave those for tomorrow as I'm a bit tired (and sore, and dizzy) atm because I had a little problem today with a bike[3], a stairway and the desire to jump the latter using the former with too much speed and little experience/skills. :-) Ah yes, the semantics of cffi-sys:foreign-var-address aren't well defined yet. Namely when the var doesn't exist different things happen with different lisps. Do you think I should work on porting the existing features to other lisps to answer your previous concerns, Kenny? Bye, [2] James nicely told me how to patch clisp for global lookup to work. I'm attaching the patch.. [3] http://article.gmane.org/gmane.lisp.steel-bank.general/813 Someone sent me a fix, which didn't make it to mailing list for some reason. (I attached it) [3] http://www.unionbike.net/MONDRAKER/imagenes1/level.jpg -- Lu?s Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www2.iro.umontreal.ca/~pinard/po/registry.cgi?team=pt -------------- next part -------------- A non-text attachment was scrubbed... Name: a.lisp Type: application/octet-stream Size: 976 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: rtld_global.patch Type: application/octet-stream Size: 517 bytes Desc: not available URL: From ktilton at nyc.rr.com Mon Jul 4 04:49:39 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Mon, 04 Jul 2005 00:49:39 -0400 Subject: [fetter-devel] Quick CFFI update In-Reply-To: <63d454e0eb25cc299a16c89b2514b861@gmail.com> References: <63d454e0eb25cc299a16c89b2514b861@gmail.com> Message-ID: <42C8BFE3.3090206@nyc.rr.com> Luis Oliveira wrote: > [Rayiner/Kenny these mails should be sent to cffi-devel only right? If > you agree, I'll be sending future mails about CFFI to cffi-devel only.] As long as Rayiner is tracking cffi-devel. I will sign up shortly. I do not think we need to worry about spamming fetter-devel yet with cffi chat. > It works with sbcl/openmcl/clisp/cmucl (actually I haven't tested with > clisp yet) but requires a patch to clisp[1] and to sbcl[2] too. > > I'm interested in your comments, and I had few questions in my notes > but I'll leave those for tomorrow as I'm a bit tired (and sore, and > dizzy) atm because I had a little problem today with a bike[3], a > stairway and the desire to jump the latter using the former with too > much speed and little experience/skills. :-) > > Ah yes, the semantics of cffi-sys:foreign-var-address aren't well > defined yet. Namely when the var doesn't exist different things happen > with different lisps. > > Do you think I should work on porting the existing features to other > lisps to answer your previous concerns, Kenny? Well, ya got me laughing over here. What happens after you have ported it to all but two Lisps and then discover it won't fly on those? What stops you from simply eyeballing the different Lisps to see if they have the necessary hooks? btw, Are all patches already determined to be necessary going to be accepted by the maintainers of the Lisp in question? Basically it sounds like you are somehow confident this story will have a happy ending, and I get the feeling that even if not, not much time will have been lost, so I am shaking my head but not worried. How long /will/ this exercise take? kt From luismbo at gmail.com Mon Jul 4 14:16:29 2005 From: luismbo at gmail.com (Luis Oliveira) Date: Mon, 4 Jul 2005 15:16:29 +0100 Subject: [fetter-devel] Quick CFFI update In-Reply-To: <42C8BFE3.3090206@nyc.rr.com> References: <63d454e0eb25cc299a16c89b2514b861@gmail.com> <42C8BFE3.3090206@nyc.rr.com> Message-ID: <048309567ac4c36e50274a462b87a415@gmail.com> On 4/jul/2005, at 05:49, Kenny Tilton wrote: > What stops you from simply eyeballing the different Lisps to see if > they have the necessary hooks? Ok, I will do this. > btw, Are all patches already determined to be necessary going to be > accepted by the maintainers of the Lisp in question? Regarding SBCL, it *is* a bug, so I'm pretty sure it'll be accepted. About clisp I'm not so sure as it's not a bug per se. Also, IIRC, other lisps might have the same issue as clisp (having to specify with library the var/function is in) so I guess I might have to add a way to specify that anyway. I think James is just trying not to specify that unless absolutely necessary. > Basically it sounds like you are somehow confident this story will > have a happy ending, and I get the feeling that even if not, not much > time will have been lost, so I am shaking my head but not worried. > > How long /will/ this exercise take? Well, I guess the UFFI approach would be safer (and easier even), but I feel this approach is better. I also have the feeling that with the low-level operators cffi exports, it'll be easier to support C++ (for Fetter too). (And James plans an Objective C bridge someday). But if you feel I'm wrong, please do say so! I'm certainly not very experienced in these kinds of decisions. -- Lu?s Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www2.iro.umontreal.ca/~pinard/po/registry.cgi?team=pt From ktilton at nyc.rr.com Mon Jul 4 15:21:25 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Mon, 04 Jul 2005 11:21:25 -0400 Subject: [fetter-devel] Quick CFFI update In-Reply-To: <048309567ac4c36e50274a462b87a415@gmail.com> References: <63d454e0eb25cc299a16c89b2514b861@gmail.com> <42C8BFE3.3090206@nyc.rr.com> <048309567ac4c36e50274a462b87a415@gmail.com> Message-ID: <42C953F5.40107@nyc.rr.com> Luis Oliveira wrote: > On 4/jul/2005, at 05:49, Kenny Tilton wrote: > >> What stops you from simply eyeballing the different Lisps to see if >> they have the necessary hooks? > > > Ok, I will do this. CFFI seems from a distance to be something for the open source crowd. "commercial lisps" are left as an exercise. Is this because it was expected one would have to hack on the implementations? Or this may be because of performance issues? ie, CMUCL needed a fix? I do not know. Anyway, any "universal" FFI has to handle popular commercial lisps. Of course we always have a hybrid approach as a fallback, and if CFFI solves a performance problem it belongs in the mix somewhere. > > > >> btw, Are all patches already determined to be necessary going to be >> accepted by the maintainers of the Lisp in question? > > > Regarding SBCL, it *is* a bug, so I'm pretty sure it'll be accepted. > About clisp I'm not so sure as it's not a bug per se. Also, IIRC, > other lisps might have the same issue as clisp (having to specify with > library the var/function is in) so I guess I might have to add a way > to specify that anyway. I think James is just trying not to specify > that unless absolutely necessary. FYI: Lispworks needs the library specified, AllegroCL does not. > > > >> Basically it sounds like you are somehow confident this story will >> have a happy ending, and I get the feeling that even if not, not much >> time will have been lost, so I am shaking my head but not worried. >> >> How long /will/ this exercise take? > > > Well, I guess the UFFI approach would be safer (and easier even), but > I feel this approach is better. I also have the feeling that with the > low-level operators cffi exports, it'll be easier to support C++ (for > Fetter too). (And James plans an Objective C bridge someday). Can you give a specific example? I am actually mentoring five projects this summer, so it helps if the issues can be laid out with specific cases, well explained. That might help your own thought processes as well. Nothing like having to explain something to someone else. > > > But if you feel I'm wrong, please do say so! I'm certainly not very > experienced in these kinds of decisions. > I just do not see the advantage yet. You mentioned the implementation-specific code was in one source file per implementation. Nice (having seen UFFI-style conditionalized code) but nothing big. Is the idea that we do an end run on the different implementations' incompatible high-level APIs by resolving differences at a lower-level, perhaps unsupported or even unavailable level? ie, UFFI creates a uniform macrofied API which expands into implementation FFIese, meaning one has to master the nuances of each high-level API and then struggle to find common ground. Whereas with CFFI at the lower level we have fewer places that can diverge (and no matter what the high-level API looks like, they are bound to support certain lowlevel capabilities)? That may all be rubbish, but it is the /kind/ of argument that would justify starting more or less from scratch on a new universal FFI. ie, I for one look for the "why?" behind your "I feel this approach is better". -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From smustudent1 at yahoo.com Mon Jul 4 19:29:59 2005 From: smustudent1 at yahoo.com (C Y) Date: Mon, 4 Jul 2005 12:29:59 -0700 (PDT) Subject: [fetter-devel] Quick CFFI update In-Reply-To: <42C953F5.40107@nyc.rr.com> Message-ID: <20050704192959.87850.qmail@web31714.mail.mud.yahoo.com> --- Kenny Tilton wrote: > > CFFI seems from a distance to be something for the open source > crowd. "commercial lisps" are left as an exercise. This isn't too surprising, is it? Commercial lisps can be expensive if you get a license that allows a wide variety of uses, and since CFFI is a free tool it would make sense it doesn't have the resources to work on the commercial lisps (at least initially.) Also, open source can be an area where new ideas and standards prove themselves, and gain in popularity to the point where the commercial guys would be interested in helping out. I for one am very hopefull that cffi, fetter, and Hello-C can finally bring an era of uniform foreign function interfacing to Common Lisp, and perhaps form the core of a revision of ANSI. > Is this because it was expected one would have to hack on the > implementations? Or this may be because of performance issues? ie, > CMUCL needed a fix? I do not know. Well, minimally, wouldn't it need extensive testing on all lisps it is intended to run on? > Anyway, any "universal" FFI has to handle popular commercial lisps. I'm thinking vendor support would make that MUCH easier - I doubt they would flock to an open source project unless they are sure it is the "winner". > Is the idea that we do an end run on the different implementations' > incompatible high-level APIs by resolving differences at a > lower-level, perhaps unsupported or even unavailable level? Wouldn't that make the most sense? I.e.: bottom layer -> implimentation specific stuff - whatever low level stuff the lisps provide themselves CFFI -> low level, using what the various lisps provide when it works and implimenting what is needed when it isn't available. CFFI should provide an interface which is both uniform and low level. Hello-C/UFFI/Fetter -> function as the top layer, the friendly layer that most programmers interact with. Not as powerful as CFFI perhaps but hiding all the nastiness of FFI interaction that can be hidden. The various lisps' high level APIs would then function more or less independantly. We "borrow" whatever features seem nice from each, but essentially state that there is the "uniform" top level api and then each implimentations high level API, those that have one. For those without, Hello-C/UFFI/Fetter becomes the ONLY high level API available. > Whereas with CFFI at the lower level we have > fewer places that can diverge (and no matter what the high-level API > looks like, they are bound to support certain lowlevel capabilities)? That makes sense to me - establish low level uniformity with CFFI, and build a new, standard layer or layers on top of it. This hides individual lisp weirdness from the user. The way this exercise makes the most sense to me is if it tries to do things the Right Way and becomes the Final solution for Lisp FFI interfacing. This is one of (perhaps the biggest, after lack of a portable, native, friendly graphics layer) problems lisp as a language faces. Yet Another FFI probably won't help much, but a final, uniform solution everyone can agree is the best way to go would help tremendously. Of course, that's just me. Cheers, CY __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From ktilton at nyc.rr.com Mon Jul 4 20:00:59 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Mon, 04 Jul 2005 16:00:59 -0400 Subject: [fetter-devel] Quick CFFI update In-Reply-To: <20050704192959.87850.qmail@web31714.mail.mud.yahoo.com> References: <20050704192959.87850.qmail@web31714.mail.mud.yahoo.com> Message-ID: <42C9957B.8020408@nyc.rr.com> C Y wrote: >--- Kenny Tilton wrote: > > >>CFFI seems from a distance to be something for the open source >>crowd. "commercial lisps" are left as an exercise. >> >> > >This isn't too surprising, is it? > I am sorry. Exactly who are you? And what is your interest here? Your comments are fine, but irrelevant to this discussion, which is why I ask. CFFI is fine just the way it is. The relevant question is: can Luis fulfill his commitment (a universal FFI) by extending CFFI, and what does he buy by taking that route. You seem to like my guess as to what he buys, but you, like me, seem to be speculating. The whole point of this discussion is to stop speculating and start making concrete, verifiable assertions of fact about CFFI and the path to true universality. The last thing Common Lisp needs is another half-baked, non-portable, and/or incomplete FFI. The whole point of these summer projects is to give CL truly universal FFI and sockets, eliminating two huge complaints interfering with more widespread acceptance and more useful open source: what good is an open source package if it uses FFI and/or sockets, hence only runs here and there? -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From jamesjb at jamesjb.com Mon Jul 4 20:08:43 2005 From: jamesjb at jamesjb.com (James Bielman) Date: Mon, 04 Jul 2005 13:08:43 -0700 Subject: [cffi-devel] Re: [fetter-devel] Quick CFFI update In-Reply-To: <42C953F5.40107@nyc.rr.com> (Kenny Tilton's message of "Mon, 04 Jul 2005 11:21:25 -0400") References: <63d454e0eb25cc299a16c89b2514b861@gmail.com> <42C8BFE3.3090206@nyc.rr.com> <048309567ac4c36e50274a462b87a415@gmail.com> <42C953F5.40107@nyc.rr.com> Message-ID: Kenny Tilton writes: > CFFI seems from a distance to be something for the open source > crowd. "commercial lisps" are left as an exercise. Is this because > it was expected one would have to hack on the implementations? Or > this may be because of performance issues? ie, CMUCL needed a fix? I > do not know. Anyway, any "universal" FFI has to handle popular > commercial lisps. Of course we always have a hybrid approach as a > fallback, and if CFFI solves a performance problem it belongs in the > mix somewhere. It's true that it's easy for me to take the long-term view about supporting more Lisp implementations since CFFI already supports the implementations that I use (mostly, I do need a LispWorks port). The reason the CFFI interface seems low-level compared to everything else is because it was designed while asking myself the question, "What is the minimum I need from a Lisp implementation in order to do useful things with its FFI?". Performance is a goal, but it is not the primary goal. If an open source implementation has gaps in this support it's relatively easy for me to write patches to add that support, and they have a fair chance of getting applied. For commercial implementations there's little else to do but ask them nicely to support this API, perhaps dealing with reasons why it's not possible for them to do it, or they don't care, or maybe have technical reasons why the interface sucks. So, I guess I took the easy way out by trying to implement things on the systems that I could be proactive about first, so when it comes time to beg others to implement them for me, I can say "but look, it works on these 5 other implementations..." Whether this is too long-term for your summer goals or not, I don't know. I'm not really trying to convince anyone of anything (although I'm certainly happy if people want to work on CFFI!), especially since I probably can't even work seriously on this for another month or so. James From smustudent1 at yahoo.com Mon Jul 4 21:01:55 2005 From: smustudent1 at yahoo.com (C Y) Date: Mon, 4 Jul 2005 14:01:55 -0700 (PDT) Subject: [fetter-devel] Quick CFFI update In-Reply-To: <42C9957B.8020408@nyc.rr.com> Message-ID: <20050704210155.70410.qmail@web31711.mail.mud.yahoo.com> --- Kenny Tilton wrote: > > C Y wrote: > > > >This isn't too surprising, is it? > > > I am sorry. Exactly who are you? And what is your interest here? > Your comments are fine, but irrelevant to this discussion, which > is why I ask. My name is Cliff Yapp. I'm essentially an observer on these lists, but I do have an interest in FFI's in Lisp from the standpoint of the Maxima computer algebra system - FFIs are considered a promising way to incorporate things like fast numerical libraries into Maxima. But for this to be viable we would need to have a univeral (at least over the free lisps we support) FFI. I apologize for being irrelevant. Since I am not currently a code contributing list member I'll refrain from further comment unless I have actual code contributions. > CFFI is fine just the way it is. The relevant question is: can Luis > fulfill his commitment (a universal FFI) by extending CFFI, and what > does he buy by taking that route. You seem to like my guess as to > what he buys, but you, like me, seem to be speculating. The whole > point of this discussion is to stop speculating and start making > concrete, verifiable assertions of fact about CFFI and the path to > true universality. Which means code. Fair enough. > The last thing Common Lisp needs is another half-baked, > non-portable, and/or incomplete FFI. The whole point of these > summer projects is to give CL truly universal FFI and sockets, > eliminating two huge complaints: interfering with more widespread > acceptance and more useful open source: what good is an open source > package if it uses FFI and/or sockets, hence only runs here and > there? Amen, Signing off, and best of luck! CY __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From luismbo at gmail.com Tue Jul 5 01:32:56 2005 From: luismbo at gmail.com (Luis Oliveira) Date: Tue, 5 Jul 2005 02:32:56 +0100 Subject: [fetter-devel] CFFI ported to Allegro CL Message-ID: <7442ddc5afdcd86999d0867f09da7ca7@gmail.com> Hello, This afternoon I ported CFFI to Allegro CL. It passes all tests on Linux/x86, fails 3 on Windows (one of them is because sqrtf doesn't exist on Windows, IIRC) and 2 on OSX (there seems to be something going on with FF:GET-ENTRY-POINT on OSX; either that, or I don't understand what get-entry-point is supposed to do). Well, yeah, there are only 32 tests so far and there are two magic numbers being passed to system::ff-funcall (I mailed Duane Rettig, I hope he'll help me figure out what those are for). Also, we are using unexported interfaces (which are however briefly mentioned in docs). But still, this was a nice start. The patch is in my darcs tree: http://common-lisp.net/project/cffi/darcs/cffi-luis -- Lu?s Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www2.iro.umontreal.ca/~pinard/po/registry.cgi?team=pt From ktilton at nyc.rr.com Tue Jul 5 02:52:18 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Mon, 04 Jul 2005 22:52:18 -0400 Subject: [cffi-devel] Re: [fetter-devel] Quick CFFI update In-Reply-To: References: <63d454e0eb25cc299a16c89b2514b861@gmail.com> <42C8BFE3.3090206@nyc.rr.com> <048309567ac4c36e50274a462b87a415@gmail.com> <42C953F5.40107@nyc.rr.com> Message-ID: <42C9F5E2.6030503@nyc.rr.com> James Bielman wrote: >Kenny Tilton writes: > > > >>CFFI seems from a distance to be something for the open source >>crowd. "commercial lisps" are left as an exercise. Is this because >>it was expected one would have to hack on the implementations? Or >>this may be because of performance issues? ie, CMUCL needed a fix? I >>do not know. Anyway, any "universal" FFI has to handle popular >>commercial lisps. Of course we always have a hybrid approach as a >>fallback, and if CFFI solves a performance problem it belongs in the >>mix somewhere. >> >> > >It's true that it's easy for me to take the long-term view about >supporting more Lisp implementations since CFFI already supports the >implementations that I use (mostly, I do need a LispWorks port). > Assuming CFFI is the right way to go in the long run, the best thing to do for the immediate objective (universal FFI now not manana) is take it as far as it will go and use native FFIs for Lisps that won't do CFFI. Then long run we have the luxury of time to jawbone non-CFFI-able Lisp maintainers. > >The reason the CFFI interface seems low-level compared to everything >else is because it was designed while asking myself the question, >"What is the minimum I need from a Lisp implementation in order to do >useful things with its FFI?". Performance is a goal, but it is not >the primary goal. > OK, but why is "the minimum necessary" a Good Thing, aside from Occam's Razor and Einstein's "as simple as possible but no simpler"? My guess is that it is easier to invent one uniform high-level API atop a few low-level FFI capabilities than take a dozen high-level APIs and come up with a consistent super-high-level API. But no one is saying that except me. And CY. :) > >If an open source implementation has gaps in this support it's >relatively easy for me to write patches to add that support, and they >have a fair chance of getting applied. > >For commercial implementations there's little else to do but ask them >nicely to support this API, perhaps dealing with reasons why it's not >possible for them to do it, or they don't care, or maybe have >technical reasons why the interface sucks. > I am anxious to get the Actual Landscape (in my mind's eye I see a 2D table, rows being Lisps, columns being needed support hooks for CFFI) to sharpen this discussion. I am also anxious, where gaps are found, for Luis to contact vendors/maintainers and get them to sign on to making necessary changes ASAP. Hopefully the prospect of being excoriated on c.l.l by Demon Kenny will bring cooperation. :) In a sense, i guess what we are doing is a Pitamnesque substandard or a CLRFI, except instead of undertaking a process a few of are undertaking some coding. I always prefer coding, myself. :) > >So, I guess I took the easy way out by trying to implement things on >the systems that I could be proactive about first, so when it comes >time to beg others to implement them for me, I can say "but look, it >works on these 5 other implementations..." > Man, I am /really/ curious about the Actual Landscape. Here is one gap: has the new killer CLisp FFI been ported to the Mac OS X version yet? Though in that case i would not consider this a gap in Hello-C/CFFI, since the old CLisp FFI was pretty weak and pretty much asked for trouble (such as being left out of portable FFI projects). > >Whether this is too long-term for your summer goals or not, I don't >know. I'm not really trying to convince anyone of anything ... > Totally understood, and no one is complaining about CFFI, which au contraire seems to be a hit. I hear even the Yobbos like it. :) But the project does have to produce a universal FFI by summer's end, so we are just trying to decide the best way. Well, let me clarify. The line I am drawing is this: if a Lisp on a platform has a sufficiently powerful native FFI, Hello-C should be extended to that Lisp and platform. CLisp on OS X is an example of a Lisp/platform that can wait until the new CLisp FFI gets ported there, and indeed we should wait, unless the CLisp folks decide to support CFFI as a way of getting an FFI on that platform. Come to think of it, that raises another question: does CFFI's high-level API rock as well as CLisp's new FFI on win32 and Unix? I am thinking of things like a lambda form being a Lisp callback from C. -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From ktilton at nyc.rr.com Tue Jul 5 03:03:07 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Mon, 04 Jul 2005 23:03:07 -0400 Subject: [fetter-devel] Quick CFFI update In-Reply-To: <20050704210155.70410.qmail@web31711.mail.mud.yahoo.com> References: <20050704210155.70410.qmail@web31711.mail.mud.yahoo.com> Message-ID: <42C9F86B.4070405@nyc.rr.com> C Y wrote: >--- Kenny Tilton wrote: > > >>C Y wrote: >> >> >>>This isn't too surprising, is it? >>> >>> >>> >>I am sorry. Exactly who are you? And what is your interest here? >>Your comments are fine, but irrelevant to this discussion, which >>is why I ask. >> >> > >My name is Cliff Yapp. I'm essentially an observer on these lists, but >I do have an interest in FFI's in Lisp from the standpoint of the >Maxima computer algebra system - FFIs are considered a promising way to >incorporate things like fast numerical libraries into Maxima. But for >this to be viable we would need to have a univeral (at least over the >free lisps we support) FFI. > >I apologize for being irrelevant. Since I am not currently a code >contributing list member I'll refrain from further comment unless I >have actual code contributions. > Hi, Cliff. I did not mean to bite your head off, just a couple of fingers. :) Everything you said was spot on as far as I am concerned, and input on design questions from the interested and FFI-savvy community is great to have, so don't hold back. -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From ktilton at nyc.rr.com Tue Jul 5 03:15:44 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Mon, 04 Jul 2005 23:15:44 -0400 Subject: [fetter-devel] CFFI ported to Allegro CL In-Reply-To: <7442ddc5afdcd86999d0867f09da7ca7@gmail.com> References: <7442ddc5afdcd86999d0867f09da7ca7@gmail.com> Message-ID: <42C9FB60.5000808@nyc.rr.com> Luis Oliveira wrote: > Hello, > > This afternoon I ported CFFI to Allegro CL. It passes all tests on > Linux/x86, fails 3 on Windows (one of them is because sqrtf doesn't > exist on Windows, IIRC) and 2 on OSX (there seems to be something > going on with FF:GET-ENTRY-POINT on OSX; either that, or I don't > understand what get-entry-point is supposed to do). > > Well, yeah, there are only 32 tests so far ... I have been meaning to bring up this point, viz, that the regression test deliverable (both sides, Lisp and C/C++) would be nice to have at this point. > and there are two magic numbers being passed to system::ff-funcall (I > mailed Duane Rettig, I hope he'll help me figure out what those are > for). Also, we are using unexported interfaces (which are however > briefly mentioned in docs). But still, this was a nice start. ugh, this is the kind of thing that scares me: a partial (?) test suite works on one OS out of three, with magic numbers and unexported (aka unsupported) hooks. (I know it was only a brief effort, but often the first 90% of a task lies down just so the last 10% can get a clean shot at me. ) With CFFI/Hello-C threatening to become a standard, will Franz want to help CFFI plant its flag atop AllegroCL? That is what happens if all of Lisp open source shifts to CFFI. What if Franz thinks their FFI is better? In that case, they will see CFFI as hurting their product, and rightfully do their best to make ineluctable their native FFI (even if delivered UFFI-like via macrology). Well, sounds like you are having fun. Keep us posted. :) -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From luismbo at gmail.com Tue Jul 5 03:19:07 2005 From: luismbo at gmail.com (Luis Oliveira) Date: Tue, 5 Jul 2005 04:19:07 +0100 Subject: [cffi-devel] Re: [fetter-devel] Quick CFFI update In-Reply-To: <42C9F5E2.6030503@nyc.rr.com> References: <63d454e0eb25cc299a16c89b2514b861@gmail.com> <42C8BFE3.3090206@nyc.rr.com> <048309567ac4c36e50274a462b87a415@gmail.com> <42C953F5.40107@nyc.rr.com> <42C9F5E2.6030503@nyc.rr.com> Message-ID: On 5/jul/2005, at 03:52, Kenny Tilton wrote: > OK, but why is "the minimum necessary" a Good Thing, aside from > Occam's Razor and Einstein's "as simple as possible but no simpler"? > My guess is that it is easier to invent one uniform high-level API > atop a few low-level FFI capabilities than take a dozen high-level > APIs and come up with a consistent super-high-level API. But no one is > saying that except me. And CY. :) And I. Well, and the more I put effort into CFFI the more it seems to be true. > Man, I am /really/ curious about the Actual Landscape. Here is one > gap: has the new killer CLisp FFI been ported to the Mac OS X version > yet? Though in that case i would not consider this a gap in > Hello-C/CFFI, since the old CLisp FFI was pretty weak and pretty much > asked for trouble (such as being left out of portable FFI projects). Heh, guess what, not only does clisp's ffi work on OSX (at least the CVS version), but it also passes all CFFI tests: Doing 32 pending tests of 32 tests total. [...] No tests failed. 0 errors, 0 warnings > Come to think of it, that raises another question: does CFFI's > high-level API rock as well as CLisp's new FFI on win32 and Unix? I am > thinking of things like a lambda form being a Lisp callback from C. Not yet, I guess that's a goal though. At least for me. -- Lu?s Oliveira http://student.dei.uc.pt/~lmoliv/ Equipa Portuguesa do Translation Project http://www2.iro.umontreal.ca/~pinard/po/registry.cgi?team=pt From jamesjb at jamesjb.com Tue Jul 5 03:37:26 2005 From: jamesjb at jamesjb.com (James Bielman) Date: Mon, 04 Jul 2005 20:37:26 -0700 Subject: [cffi-devel] Re: [fetter-devel] Quick CFFI update In-Reply-To: <42C9F5E2.6030503@nyc.rr.com> (Kenny Tilton's message of "Mon, 04 Jul 2005 22:52:18 -0400") References: <63d454e0eb25cc299a16c89b2514b861@gmail.com> <42C8BFE3.3090206@nyc.rr.com> <048309567ac4c36e50274a462b87a415@gmail.com> <42C953F5.40107@nyc.rr.com> <42C9F5E2.6030503@nyc.rr.com> Message-ID: Kenny Tilton writes: > CLisp on OS X is an example of a Lisp/platform that can wait until > the new CLisp FFI gets ported there, and indeed we should wait, > unless the CLisp folks decide to support CFFI as a way of getting an > FFI on that platform. I'm not aware that the OS X FFI is any different from the FFI on other platforms. I've used the CLISP dynamic FFI just fine on OS X. James From jamesjb at jamesjb.com Tue Jul 5 05:13:24 2005 From: jamesjb at jamesjb.com (James Bielman) Date: Mon, 04 Jul 2005 22:13:24 -0700 Subject: [cffi-devel] Re: [fetter-devel] Quick CFFI update In-Reply-To: <42C9F5E2.6030503@nyc.rr.com> (Kenny Tilton's message of "Mon, 04 Jul 2005 22:52:18 -0400") References: <63d454e0eb25cc299a16c89b2514b861@gmail.com> <42C8BFE3.3090206@nyc.rr.com> <048309567ac4c36e50274a462b87a415@gmail.com> <42C953F5.40107@nyc.rr.com> <42C9F5E2.6030503@nyc.rr.com> Message-ID: Kenny Tilton writes: > My guess is that it is easier to invent one uniform high-level API > atop a few low-level FFI capabilities than take a dozen high-level > APIs and come up with a consistent super-high-level API. But no one > is saying that except me. And CY. :) Err, if I understand you right this is exactly what CFFI does! For example, let's define the C timeval structure using CFFI: (defcstruct timeval (tv-sec :long) (tv-usec :long)) The macroexpansion of this definition is 100% ANSI CL. No calls to the host FFI are made, it just defines a CLOS instance associated with TIMEVAL containing the layout of the slots in the TIMEVAL structure. (Actually that's not entirely true, it has to ask the Lisp FFI for the size and alignment of the :LONG type, but that's not very difficult). Now, let's allocate a TIMEVAL on the heap: (let ((tv (foreign-object-alloc 'timeval))) ...) FOREIGN-OBJECT-ALLOC is a function (again, ANSI CL) that ends up using the CFFI-SYS (implementation layer) interface, to allocate enough bytes to hold the structure: (foreign-alloc 8) Which ends up calling some sort of MALLOC function defined by the host FFI (every FFI I've seen exports this primitive already). If we set a slot in TV: (setf (foreign-slot-value tv 'timeval 'tv-sec) 100) FOREIGN-SLOT-VALUE is another portable function (but will be inlined via a compiler-macro since the types are constant), that uses a CFFI-SYS primitive to actually frob the memory: (setf (cffi-sys:%mem-ref tv :long 0) 100) %MEM-REF is a more complicated primitive; this expression is equivalent to: *(((long *) tv) + 0) = 100; and is also inlined via compiler macros to, for example, on SBCL: (setf (sb-sys:sap-ref-32 tv 0) 100) Again, every FFI I've looked at so far has this primitive in some form or another (CLISP didn't have a fast one, although I think one is being worked on). Most of the CFFI-SYS primitives are like this, just frobbing memory and doing operations on pointers, and there doesn't seem to be too much trouble porting these. There are some more complex primitives for doing block memory transfers and getting pointers to the data in Lisp vectors, but these can be implemented in terms of other primitives if necessary. The only primitive that is really problematic is FOREIGN-FUNCALL, which is like OpenMCL's EXTERNAL-CALL macro. It provides a method for calling a foreign function without using a declarative interface. For example: ? (foreign-funcall "sqrtf" :float 16.0 :float) The last argument to FOREIGN-FUNCALL is the return type, preceding keyword/value pairs are the types and values of the function arguments. I like to short-circuit the host FFI's declarative interface to functions because this macro makes it very easy to deal with varargs functions (which I need to write the portable Objective-C bridge I want to write down the road). Now, users (well, hypothetical users anyway) don't generally use this directly, they use DEFCFUN, which is like UFFI:DEF-FUNCTION: (defcfun "sqrtf" :float (x :float)) If there's a compromise to be made, this is where I would suggest making it. Hello-C could redefine DEFCFUN to macroexpand into the host FFI's declarative interface (DEF-FOREIGN-CALL, etc), instead of using FOREIGN-FUNCALL. (It would still need to invoke any type translators defined in the argument and result types, to do things like convert Lisp strings to/from C strings). I suspect that, making this compromise, it would be easy to support every reasonable host FFI for the Lisps you are interested in. I suggest that you look over Luis' Allegro CL port of the CFFI-SYS interface at: http://common-lisp.net/project/cffi/darcs/cffi-luis/src/cffi-allegro.lisp The interface is really not all that big, and apart from FOREIGN-FUNCALL it's pretty simple stuff. So, to get back to your original question, what we gain with a minimal interface is we can define a little bit of core functionality and get everything else "for free". > I am anxious to get the Actual Landscape (in my mind's eye I see a > 2D table, rows being Lisps, columns being needed support hooks for > CFFI) to sharpen this discussion. I am also anxious, where gaps are > found, for Luis to contact vendors/maintainers and get them to sign > on to making necessary changes ASAP. Hopefully the prospect of being > excoriated on c.l.l by Demon Kenny will bring cooperation. :) Hopefully I've been able to convince you somewhat that the interface really isn't all that complicated, and the hooks needed are pretty basic (and likely to already exist), except for FOREIGN-FUNCALL which could be done differently to ease development. > In a sense, i guess what we are doing is a Pitamnesque substandard > or a CLRFI, except instead of undertaking a process a few of are > undertaking some coding. I always prefer coding, myself. :) Basically, although I think there's a much better chance of getting things to happen when you can enter the discussion with a working implementation, instead of arguing over some specification. And now I'm going up to the roof to watch the fireworks over Puget Sound. :) James From ktilton at nyc.rr.com Tue Jul 5 05:22:31 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Tue, 05 Jul 2005 01:22:31 -0400 Subject: [cffi-devel] Re: [fetter-devel] Quick CFFI update In-Reply-To: References: <63d454e0eb25cc299a16c89b2514b861@gmail.com> <42C8BFE3.3090206@nyc.rr.com> <048309567ac4c36e50274a462b87a415@gmail.com> <42C953F5.40107@nyc.rr.com> <42C9F5E2.6030503@nyc.rr.com> Message-ID: <42CA1917.80204@nyc.rr.com> Luis Oliveira wrote: > On 5/jul/2005, at 03:52, Kenny Tilton wrote: > >> Man, I am /really/ curious about the Actual Landscape. Here is one >> gap: has the new killer CLisp FFI been ported to the Mac OS X version >> yet? Though in that case i would not consider this a gap in >> Hello-C/CFFI, since the old CLisp FFI was pretty weak and pretty much >> asked for trouble (such as being left out of portable FFI projects). > > > Heh, guess what, not only does clisp's ffi work on OSX (at least the > CVS version), but it also passes all CFFI tests: A little more precision is required. CFFI does not go thru an implementation's exported, documented, high-level FFI, it goes thru lower-level hooks used by that implementation's exported. documented, supported FFI. Youse guys seem to have started referring to the low-level, unexported, undocumented, unsopported, internal hooks as "CLisp's FFI". So whaddyawannado to make communication possible? Possibly we have three FFIs: FFI/hi, FFI/lo (low-level but exported/supported), and FFI/internals (unsupported). I might think from your report and James's that CLisp had finished porting its FFI/hi to OS X, but since CFFI does not go thru anybody's FFI/hi I am guessing you mean a usable FFI/lo or FFI/internals was always there to support CFFI. This, btw, would be a strong argument for CFFI: even if CLisp is closing the FFI/hi gap, the fact remains that CFFI would have made that effort unnecessary. >> Come to think of it, that raises another question: does CFFI's >> high-level API rock as well as CLisp's new FFI on win32 and Unix? I >> am thinking of things like a lambda form being a Lisp callback from C. > > > Not yet, I guess that's a goal though. At least for me. I am not sure where I come down on this as a requirement. Without it, some Lisps' native FFIs have at least some clear added value over CFFI. But I had to offer a poor man's subsitute when I ported Cells-Gtk from CLisp FFI to UFFI, using a scintilla of new syntax and some diabolical macrology, and I do not feel much of the power of lambda qua callback was lost. -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From smustudent1 at yahoo.com Tue Jul 5 06:09:05 2005 From: smustudent1 at yahoo.com (C Y) Date: Mon, 4 Jul 2005 23:09:05 -0700 (PDT) Subject: [fetter-devel] Quick CFFI update In-Reply-To: <42C9F86B.4070405@nyc.rr.com> Message-ID: <20050705060905.47118.qmail@web31703.mail.mud.yahoo.com> --- Kenny Tilton wrote: > > Hi, Cliff. I did not mean to bite your head off, just a couple of > fingers. :) Everything you said was spot on as far as I am concerned, > and input on design questions from the interested and FFI-savvy > community is great to have, so don't hold back. > > -- > Kenny No worries :-). Irrelevant discussion is often a severe drag on getting stuff done, so the request for ID was quite reasonable. I'm not currently all that FFI savvy - more a case of trying to visualize what would be useful in Maxima, which isn't really a design criteria ;-). Which does bring up one potentially useful question I can pose - for those without much FFI experience, but wanting to get up to speed so as to be actually useful to this effort, what are the recommended resources? Books, implimentation docs, FFI user guides? Perhaps a couple "read these before posting" links would divert the newbies and convert them into slightly less new newbies ;-). Cheers, CY ____________________________________________________ Yahoo! Sports Rekindle the Rivalries. Sign up for Fantasy Football http://football.fantasysports.yahoo.com From ktilton at nyc.rr.com Tue Jul 5 15:13:17 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Tue, 05 Jul 2005 11:13:17 -0400 Subject: [fetter-devel] Re: [cffi-devel] Quick CFFI update In-Reply-To: References: <63d454e0eb25cc299a16c89b2514b861@gmail.com> <42C8BFE3.3090206@nyc.rr.com> <048309567ac4c36e50274a462b87a415@gmail.com> <42C953F5.40107@nyc.rr.com> <42C9F5E2.6030503@nyc.rr.com> <42CA1B86.9040805@nyc.rr.com> Message-ID: <42CAA38D.1080804@nyc.rr.com> James Bielman wrote: >Kenny Tilton writes: > > > >>Great, and all I am saying is that Luis can fire up a Lisp, bang >>away with APROPOS for a few minutes, consult a Lisp's doc, and allow >>us to delete "likely to" above. :) >> >> > >I went ahead and started a matrix for implementation support at: > > http://common-lisp.net/project/cffi > Brilliant! Thanks, I know your time is scarce just now. Now from the Department of Give Them An Inch And They'll Want A Mile: Don't we need separate columns for different OSes? And I would like to see differentiation between "works" and "works, but using internals which might change in the next release so look out". -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From ktilton at nyc.rr.com Tue Jul 5 15:22:49 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Tue, 05 Jul 2005 11:22:49 -0400 Subject: [fetter-devel] Quick CFFI update In-Reply-To: <20050705060905.47118.qmail@web31703.mail.mud.yahoo.com> References: <20050705060905.47118.qmail@web31703.mail.mud.yahoo.com> Message-ID: <42CAA5C9.1050905@nyc.rr.com> C Y wrote: >--- Kenny Tilton wrote: > > >>Hi, Cliff. I did not mean to bite your head off, just a couple of >>fingers. :) Everything you said was spot on as far as I am concerned, >>and input on design questions from the interested and FFI-savvy >>community is great to have, so don't hold back. >> >>-- >>Kenny >> >> > >No worries :-). Irrelevant discussion is often a severe drag on >getting stuff done, > The funny thing is that I also want project developers to get input from the community via c.l.l, Planet Lisp, and #Lisp. Talk about a potential for irrelevant discussion.... :) >Which does bring up one potentially useful question I can pose - for >those without much FFI experience, but wanting to get up to speed so as >to be actually useful to this effort, what are the recommended >resources? Books, implimentation docs, FFI user guides? > I think you can grab the AllegroCL Trial for any OS and get their FFI doc and discussion. That might even be accessible via the Web. That is the one I know, but likewise for Lispworks or even CLisp. I think just reading that will give you a feel for the issues as an applications person. If you are really feeling energetic, try using the FFI of your implementation to get to a fun library, such as OpenGL or SDL. You can cheat when you get stuck by finding existing bindings on c-l.net. A dead simple library is OpenAL, by the way, for (pretty much) playing WAV files with 3D effects such as Doppler shift. Mind you, our joint objective is sparing application developers most of this grief. We'll call it "C/C++ Plug 'n Play", assuming no one has already used that tag line. :) -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From smustudent1 at yahoo.com Tue Jul 5 18:00:21 2005 From: smustudent1 at yahoo.com (C Y) Date: Tue, 5 Jul 2005 11:00:21 -0700 (PDT) Subject: [cffi-devel] Re: [fetter-devel] Quick CFFI update In-Reply-To: <42CAA5C9.1050905@nyc.rr.com> Message-ID: <20050705180021.55830.qmail@web31709.mail.mud.yahoo.com> --- Kenny Tilton wrote: > > > >No worries :-). Irrelevant discussion is often a severe drag on > >getting stuff done, > > > The funny thing is that I also want project developers to get input > from the community via c.l.l, Planet Lisp, and #Lisp. Talk about a > potential for irrelevant discussion.... :) Oh, my. ;-) Well, no question there are a lot of smart people there. > I think you can grab the AllegroCL Trial for any OS and get > their FFI doc and discussion. That might even be accessible via > the Web. That is the one I know, but likewise for Lispworks or > even CLisp. I think justreading that will give you a feel for the > issues as an applications person. Here's what I've dug up so far (links provided for the lazy, which will be me when I get the time to read all this): Allegro CL 7.0 Foreign Function Interface http://www.franz.com/support/documentation/7.0/doc/foreign-functions.htm I think this is also relevant? Foreign Types http://www.franz.com/support/documentation/7.0/doc/ftype.htm Lispworks: http://www.lispworks.com/documentation/lw445/FLI/html/fli.htm It is not immediately clear if there are OS specific docs - http://www.lispworks.com/documentation/index.html Clisp: http://clisp.cons.org/impnotes/dffi.html - is this the new spiffy one? I had heard there was a new, very nice Clisp FFI but I'm not sure how that relates to online docs. This might also be of interest: http://clisp.cons.org/impnotes/foreign-pointers.html Presumably these are also worth a look from UFFI: UFFI Reference Guide http://uffi.b9.com/manual/ The ins and outs of foreign interface design http://uffi.b9.com/FFI-Design-v0.4.pdf Design Issues for Foreign Function Interfaces http://xarch.tu-graz.ac.at/autocad/lisp/ffis.html and of course: CFFI http://common-lisp.net/project/cffi/darcs/cffi/doc/cffi-sys-specification.html http://common-lisp.net/project/cffi/darcs/cffi/doc/manual.html http://common-lisp.net/project/cffi/darcs/cffi/doc/mem-vector.txt http://common-lisp.net/project/cffi/darcs/cffi/doc/shareable-vectors.txt > If you are really feeling energetic, try using the FFI of > your implementation to get to a fun library, such as OpenGL or > SDL. You can cheat when you get stuck by finding existing bindings > on c-l.net. A dead simple library is OpenAL, by the way, for > (pretty much) playing WAV files with 3D effects such as Doppler > shift. Sounds good. > Mind you, our joint objective is sparing application developers most > of this grief. We'll call it "C/C++ Plug 'n Play", assuming no one > has already used that tag line. :) Nice. Oh, I had a question about Lilac (is that on topic or reserved for later?) Auto-generation of bindings would be a godsend for people wanting to use FFIs, but the first thought that came to my mind was "how do I know what's available to be called, and what call syntax got auto-generated?" Is this a) something the user will have to figure out, based on knowledge of the C/C++ library b) logically deducable if one knows the general methods Lilac will use or c) auto-generatable using something like Albert to absorb any C++ comments and attach them to auto-generated documentation of each resulting lisp functions arguments and limitations? (Ignore this question if it is not (yet) on topic.) Cheers, CY __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From ktilton at nyc.rr.com Tue Jul 5 18:14:20 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Tue, 05 Jul 2005 14:14:20 -0400 Subject: [cffi-devel] Re: [fetter-devel] Quick CFFI update In-Reply-To: <20050705180021.55830.qmail@web31709.mail.mud.yahoo.com> References: <20050705180021.55830.qmail@web31709.mail.mud.yahoo.com> Message-ID: <42CACDFC.6030104@nyc.rr.com> C Y wrote: >--- Kenny Tilton wrote: > > >>>No worries :-). Irrelevant discussion is often a severe drag on >>>getting stuff done, >>> >>> >>> >>The funny thing is that I also want project developers to get input >>from the community via c.l.l, Planet Lisp, and #Lisp. Talk about a >>potential for irrelevant discussion.... :) >> >> > >Oh, my. ;-) Well, no question there are a lot of smart people there. > That is my thinking. The expected noise can simply be ignored, a feature of Usenet too few take advantage of. >Here's what I've dug up so far (links provided for the lazy, which will >be me when I get the time to read all this): > That's the stuff. >Clisp: >http://clisp.cons.org/impnotes/dffi.html - is this the new spiffy one? > Yes. >Nice. Oh, I had a question about Lilac (is that on topic or reserved >for later?) > Lilac got beat out by Fetter, which will be doing the autogen thing. > Auto-generation of bindings would be a godsend for people >wanting to use FFIs, but the first thought that came to my mind was >"how do I know what's available to be called, and what call syntax got >auto-generated?" > Fetter will generate a Lisp source file of bindings, so you can just peruse that to see how the Lisp turned out. But, yes, in the end the documentation of the C library will be what you need. What gets interesting is how things like booleans are handled. The C zero/non-zero becomes the Lisp nil/non-nil and vice versa. What will likely happen is that the FFI will generate two entry points, one without auto-conversion of booleans and strings, etc, and one with. So anyone using these tools to get to C/C++ libraries will need to read a little doc to get up to speed on that. >using something like Albert to absorb any C++ comments and attach them >to auto-generated documentation of each resulting lisp functions >arguments and limitations? > I for one did think of that, but I am not sure how well Albert (whatever that is) output would meld with gccxml output. This might be an exercise for another day. -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From smustudent1 at yahoo.com Tue Jul 5 18:35:10 2005 From: smustudent1 at yahoo.com (C Y) Date: Tue, 5 Jul 2005 11:35:10 -0700 (PDT) Subject: [cffi-devel] Re: [fetter-devel] Quick CFFI update In-Reply-To: <42CACDFC.6030104@nyc.rr.com> Message-ID: <20050705183511.92063.qmail@web31704.mail.mud.yahoo.com> --- Kenny Tilton wrote: > Lilac got beat out by Fetter, which will be doing the autogen thing. So Fetter is not just C++? Whoops - time to re-read the project profile. > Fetter will generate a Lisp source file of bindings, so you can just > peruse that to see how the Lisp turned out. But, yes, in the end the > documentation of the C library will be what you need. What gets > interesting is how things like booleans are handled. The C > zero/non-zero becomes the Lisp nil/non-nil and vice versa. What > will likely happen is that the FFI will generate two entry points, > one without auto-conversion of booleans and strings, etc, and one > with. So anyone using these tools to get to C/C++ libraries will > need to read a little doc to get up to speed on that. Hmm. OK. I take it then that in some cases auto-conversion is undesirable? > I for one did think of that, but I am not sure how well Albert > (whatever that is) output would meld with gccxml output. This > might be an exercise for another day. Right. (Just for reference, Albert is a documentation generation program written for lisp - home page is http://albert.sf.net) Cheers, CY ____________________________________________________ Yahoo! Sports Rekindle the Rivalries. Sign up for Fantasy Football http://football.fantasysports.yahoo.com From rayiner at gmail.com Sat Jul 9 22:48:09 2005 From: rayiner at gmail.com (Rayiner Hashem) Date: Sat, 9 Jul 2005 18:48:09 -0400 Subject: [fetter-devel] Fetter Update #1 Message-ID: Hello everyone. I'm just e-mailing to update everyone on the status of fetter. The design document took me a little bit longer than I expected. In any case, I've attached a PDF of the document to this e-mail, and I welcome any commentary on the contents. Specifically, I'm interested in ideas about how to make the input file syntax more powerful and useful, and ideas about integration of C/C++ libraries into Lisp code. The front-end stuff, like the IR and UFFI back-end is pretty well-specified, but the section on the input file and the handling of C++ classes is a bit light. For the moment, I want to concentrate of binding to a C library first, and when that works I'll flesh out and start work on the C++ stuff. Forward-looking statements: The design document marks my first milestone. I've got a pretty good start on parsing the GCC-XML file into IR, which was designed based on my experience of parsing it, so over the next week I'll write the backend stuff and try to hit my second milestone (binding to a C library) on time. Sincerely, Rayiner Hashem -------------- next part -------------- A non-text attachment was scrubbed... Name: design.pdf Type: application/pdf Size: 81006 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: design.tex Type: application/x-tex Size: 25281 bytes Desc: not available URL: From ktilton at nyc.rr.com Sun Jul 10 03:02:57 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Sat, 09 Jul 2005 23:02:57 -0400 Subject: [fetter-devel] Fetter Update #1 In-Reply-To: References: Message-ID: <42D08FE1.8010602@nyc.rr.com> Rayiner Hashem wrote: >Hello everyone. I'm just e-mailing to update everyone on the status of >fetter. The design document took me a little bit longer than I >expected. In any case, I've attached a PDF of the document to this >e-mail, and I welcome any commentary on the contents. > The first impression is that it is well thought out. A minor quibble: you say defbinding is analogous to defpackage, but do not make explicit the obvious, viz, that the first argument to defbinding becomes the package name. I mean, it is totally obvious, but you might want to make it explicit. Now I could answer this myself, but I have somehow managed to lose the original fetter proposal: is support for VC++ planned? i think I recall that it was, but it's hazy. So as an aside, please re-send the original proposal to me. Or maybe put it up on the FTP location for your fetter project on cl.net and link to it from the Web page for fetter. You might also put this dsign PDF up there and link to it. Because.... ... I think greater community involvement/feedback in these vital projects might help. You seem to have a pretty clear idea where you are going and, hey, it's Lisp, it is easy to revise as new or better ideas come along, but there are some smart Lisp people out there who might have some good ideas. So I am thinking an announcement posted a couple of places inviting people to track fetter (and all the Lisp NYC projects) might be a good idea. if the discussion itself stays on the fetter mailing list, I think we can avoid the nonsense i would expect from a discussion on c.l.l itself. > Specifically, >I'm interested in ideas about how to make the input file syntax more >powerful and useful, and ideas about integration of C/C++ libraries >into Lisp code. > Heh, I just wait for things to go wrong or come up short and then refactor. :) Do you have a C++ library in mind as a guinea pig? Something I thought would be useful is to create a test C++ library which exercised all the capabilities of fetter and hello-c. This would serve as regression test and even documentation, since folks suing fetter to get to c++ would be able to see what Lisp was used to get to each C++ component, data structure or function. But that could be a big job. Just a thought. -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page From rayiner at gmail.com Sun Jul 10 05:44:18 2005 From: rayiner at gmail.com (Rayiner Hashem) Date: Sun, 10 Jul 2005 01:44:18 -0400 Subject: [fetter-devel] Fetter Update #1 In-Reply-To: <42D08FE1.8010602@nyc.rr.com> References: <42D08FE1.8010602@nyc.rr.com> Message-ID: > A minor quibble: you say defbinding is analogous to defpackage, but do > not make explicit the obvious, viz, that the first argument to > defbinding becomes the package name. I mean, it is totally obvious, but > you might want to make it explicit. Good catch. There are a couple of things like this I need to comb through. I'll post updated versions as I progress through the project, so I'll update that for the next one. > Now I could answer this myself, but I have somehow managed to lose the > original fetter proposal: is support for VC++ planned? I wasn't originally planning on it, since I don't have copies of either Windows or Visual C++. I suppose the Windows userbase isn't too small to be worth supporting, though (tongue firmly in cheek), so I could do it once I've fulfilled the SoC requirements. GCC-XML will give us support for VC++ "for free" when dealing with C libraries. What will need to be specialized for Visual C++ is the utility library, which will have to be modified to understand the COM ABI v-table format (the virtual table format for VC++). > i think I recall > that it was, but it's hazy. So as an aside, please re-send the original > proposal to me. I've attached a copy of the PDF version. I've created a project page for fetter that has links to these documents. I'll upload that as soon as I figure out how... > ... I think greater community involvement/feedback in these vital > projects might help. You seem to have a pretty clear idea where you are > going and, hey, it's Lisp, it is easy to revise as new or better ideas > come along, but there are some smart Lisp people out there who might > have some good ideas. So I am thinking an announcement posted a couple > of places inviting people to track fetter (and all the Lisp NYC > projects) might be a good idea. if the discussion itself stays on the > fetter mailing list, I think we can avoid the nonsense i would expect > from a discussion on c.l.l itself. I agree. Do you have any suggestions? I don't really know where folks interested in this "hang out". > Heh, I just wait for things to go wrong or come up short and then > refactor. :) Good point. I'm trying to do this project in a more "Lisp style", but I have to fight my C++-induced proclivity to overplan things... > > Do you have a C++ library in mind as a guinea pig? For milestone #2 (C code), I had SDL in mind, and for milestone #3, maybe FLTK. I wanted a GUI library, because it gives a visual demo, but I realized Qt would be a problem because I'd have to deal with the preprocessor. I'd eventually like to get Qt working, though. > Something I thought > would be useful is to create a test C++ library which exercised all the > capabilities of fetter and hello-c. This would serve as regression test > and even documentation, since folks suing fetter to get to c++ would be > able to see what Lisp was used to get to each C++ component, data > structure or function. But that could be a big job. Just a thought. Not too big, I think. I've got a pretty good idea of the special cases involved. Basically, you've got 3 cases for function calls, and 4 cases for class object layouts, so a couple of dozen test items could probably cover a lot of useful ground. Sincerely, Rayiner Hashem > > -- > Kenny > > Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film > > "If you plan to enter text which our system might consider to be obscene, check here to certify that you are old enough to hear the resulting output." -- Bell Labs text-to-speech interactive Web page > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: proposal.pdf Type: application/pdf Size: 38777 bytes Desc: not available URL: From ktilton at nyc.rr.com Tue Jul 19 18:57:46 2005 From: ktilton at nyc.rr.com (Kenny Tilton) Date: Tue, 19 Jul 2005 14:57:46 -0400 Subject: [fetter-devel] Re: Progress Report for Week of July 18 In-Reply-To: References: Message-ID: <42DD4D2A.5030809@nyc.rr.com> Rayiner, Thanks. Looks like steady progress. Any interesting obstacles? cheers, kenny Rayiner Hashem wrote: >Progress Report for Fetter Project, Week of July 18 > >Progress Towards Milestone 2: > >- Frontend > - Reading of (primitive) configuration file specified in design document > - Parsing of GCC-XML file into IR as specified in design document > - Implemented simplification passes required for handling C code >- Backend > - UFFI declaration generation for part of the IR >- Utility Library > - Looked into how VC++'s ABI compared to GCC's, for eventual VC support >- Website > - Finished (primitive) front-page, but can't log into account >(can't decrypt account info) > - Contacted Mr. Baringer for account info, waiting. >- Advertising > - Waiting to hit milestone 2 and put up the website before contacting people > >Todo for Milestone 2: >- Backend > - Support rest of the IR required for C code >- Website > - Upload website >- Advertising > - Post messages in strategic locations outlined by Mr. Tilton > >Sincerely, > Rayiner Hashem > > > > -- Kenny Why Lisp? http://lisp.tech.coop/RtL%20Highlight%20Film "I've wrestled with reality for 35 years, Doctor, and I'm happy to state I finally won out over it." Elwood P. Dowd, "Harvey", 1950 From rayiner at gmail.com Tue Jul 19 18:06:57 2005 From: rayiner at gmail.com (Rayiner Hashem) Date: Tue, 19 Jul 2005 14:06:57 -0400 Subject: [fetter-devel] Progress Report for Week of July 18 Message-ID: Progress Report for Fetter Project, Week of July 18 Progress Towards Milestone 2: - Frontend - Reading of (primitive) configuration file specified in design document - Parsing of GCC-XML file into IR as specified in design document - Implemented simplification passes required for handling C code - Backend - UFFI declaration generation for part of the IR - Utility Library - Looked into how VC++'s ABI compared to GCC's, for eventual VC support - Website - Finished (primitive) front-page, but can't log into account (can't decrypt account info) - Contacted Mr. Baringer for account info, waiting. - Advertising - Waiting to hit milestone 2 and put up the website before contacting people Todo for Milestone 2: - Backend - Support rest of the IR required for C code - Website - Upload website - Advertising - Post messages in strategic locations outlined by Mr. Tilton Sincerely, Rayiner Hashem From rayiner at gmail.com Sat Jul 30 21:12:14 2005 From: rayiner at gmail.com (Rayiner Hashem) Date: Sat, 30 Jul 2005 17:12:14 -0400 Subject: [fetter-devel] Re: Ping Summer of Code In-Reply-To: <42EA81E3.7050102@nyc.rr.com> References: <42E7D197.7090702@nyc.rr.com> <42EA81E3.7050102@nyc.rr.com> Message-ID: Okay, I've packaged up the frontend and modified the driver a bit to print the IR. The output isn't pretty, but its fairly complete. Usage is fairly simple. Make sure you have the dependencies. I'm using Debian unstable on amd64 SBCL 0.9.2 for x86-64 cl-s-xml (s-xml XML parser) cl-asdf (ASDF) cl-split-sequence (sequence splitting utility) GCC-XML 0.7-CVS (current CVS version) Just uncompress and unpack the tarball, cd into the main directory, and type 'sbcl --load testsuite/test-basic.lisp'. This will run the basic test script I've been using, which parses the OpenGL headers. There is a file testsuite/opengl.binding that holds the config file for that test. You can modify the script and the binding file to test other things if you want. It should be robust, but the only things I've tested it on so far is SDL, OpenGL, and the little test-simple.c file I have in the testsuite directory. From an SBCL prompt, you can also do 'asdf:operate 'asdf:load-op 'fetter' then run 'frontend:create-binding ". A little description of the source tree: fetter - the main program fetter/frontend - the frontend code driver.lisp - The overall driver program (like a compiler driver) frontend.lisp - Various definitions for the front-end. Contains the IR classes. package.lisp - Definition of the 'frontend' package parser.lisp - The GCC-XML parser simplifier.lisp - The simplification passes utility.lisp - Various utility routines for handling the L-XML format The parser is the interesting part, so its worth describing here. The main function is parse-gccxml-output, which reads the XML file then calls convert-xml-nodes to convert it to the IR. The conversion process works as follows: - There are three main data structures, the ir-index and the xml-index. Each is a hash-table indexed by GCC-XML 'id' attribute and mapping to either an XML node or an IR node. The edge-queue is a vector containing a list of edges that should be added to the IR graph once all the nodes are constructed. This delayed construction of edges is necessary because GCC-XML nodes aren't in any particular order. Thus, when a given IR node is being constructed, it can't count on any nodes its definition refers to already existing in the ir-index. - The basic control flow is the dolist in convert-xml-nodes. It goes through each top-level XML node in the GCC-XML tree, looks up the type of the element ('Function' or 'PointerType') and calls the appropriate handler function. The handler map is defined in the big assoc list at the top of the file. Each of the handler functions create an IR node corresponding to the XML node, and queues any edges that will exist in the graph in the edge-queue. - Once all the nodes are converted and entered into the index, the function build-graph connects all the nodes with the edges specified in edge-queue, creates a root node, and adds an edge from that to the root namespace. It then returns the root node. - The indices (ir-index and xml-index) are not exposed to anything outside of the parser. Their view of the IR is just a pointer to the root node. Well, that's that. Please contact me if you have trouble with anything. Sincerely, Rayiner Hashem On 7/29/05, Kenny Tilton wrote: > > > Rayiner Hashem wrote: > > >Sorry for being dodgy with the e-mails the last several days. I'm kind > >of in a finals crunch and have been almost living in the library. I'll > >be 100% free after the beginning of next week as soon as they're over. > >In any case, I'll send you and Luis a copy of the front-end as soon as > >I have time to go home and package things up. Probably Friday evening, > >after classes and review sessions... > > > OK. Keep me posted. Thx. > > kt > > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: fetter-frontend.tar.gz Type: application/x-gzip Size: 85821 bytes Desc: not available URL: