From edi at agharta.de Wed Dec 27 12:48:17 2006 From: edi at agharta.de (Edi Weitz) Date: Wed, 27 Dec 2006 13:48:17 +0100 Subject: [flexi-streams-devel] New release 0.9.0 Message-ID: ChangeLog: Version 0.9.0 2006-12-27 Complete re-factoring to improve performance and reduce consing (at least for LispWorks) Added some tests Added *PROVIDE-USE-VALUE-RESTART* Added FLEXI-STREAM-POSITION-SPEC-ERROR condition Download: http://weitz.de/files/flexi-streams.tar.gz SBCL (and maybe CMUCL) users please note this: http://thread.gmane.org/gmane.lisp.steel-bank.general/1400/ From edi at agharta.de Wed Dec 27 20:40:28 2006 From: edi at agharta.de (Edi Weitz) Date: Wed, 27 Dec 2006 21:40:28 +0100 Subject: [flexi-streams-devel] New release 0.9.1 In-Reply-To: (Edi Weitz's message of "Wed, 27 Dec 2006 13:48:17 +0100") References: Message-ID: ChangeLog: Version 0.9.1 2006-12-27 More performance improvements (thanks to Robert J. Macomber for SBCL hints) Download: http://weitz.de/files/flexi-streams.tar.gz From vodonosov at mail.ru Wed Dec 27 20:54:07 2006 From: vodonosov at mail.ru (Anton Vodonosov) Date: Wed, 27 Dec 2006 22:54:07 +0200 Subject: [flexi-streams-devel] New release 0.9.0 In-Reply-To: References: Message-ID: <4592DD6F.5020200@mail.ru> Hi Edi, Instead of creating a class for each external format in order to optimize STREAM-READ-CHAR, we can just take all decision making code out from STREAM-READ-CHAR to (SETF FLEXI-STREAM-EXTERNAL-FORMAT). Example of how it may be implemented in the bottom of the letter (only character encoding handling is refactored for this demonstration, newline encoding is leaved unchanged). I've tested it on clisp 2.41 on my Windows machine. Function FOO from the testbench code you posted to http://thread.gmane.org/gmane.lisp.steel-bank.general/1400/ handles ntoskrnl.exe file with different flexi-streams versions as follows: 8.0 - 151 sec 9.0 - 40 sec 8.0 with my changes - 79 sec (note, only character encoding handling was optimized here) Regards, -Anton (defclass flexi-input-stream (flexi-stream fundamental-binary-input-stream fundamental-character-input-stream) ( .... (get-char-code-func :initform nil)) ... ) (defmethod stream-read-char ((stream flexi-input-stream)) ... ;(let ((char-code (get-char-code))) (let ((char-code (funcall (slot-value stream 'get-char-code-func) stream))) ... ) #-:lispworks (defmethod initialize-instance :after ((flexi-stream flexi-input-stream) &rest initargs) (setf (slot-value flexi-stream 'get-char-code-func) (create-char-reader flexi-stream (external-format-name (flexi-stream-external-format flexi-stream))))) (defmethod (setf flexi-stream-external-format) :after (new-value (stream flexi-input-stream)) (setf (slot-value stream 'get-char-code-func) (create-char-reader stream (external-format-name new-value)))) (defun create-char-reader (stream external-format-name) (format t "create-char-reader, stream: ~A, external-format-name: ~A~%" stream external-format-name) (cond ((ascii-name-p external-format-name) #'(lambda (stream) (read-char-8-bit stream +ascii-table+))) ((koi8-r-name-p external-format-name) #'(lambda (stream)(read-char-8-bit stream +koi8-r-table+))) ((iso-8859-name-p external-format-name) (let ((table (cdr (assoc external-format-name +iso-8859-tables+ :test #'eq)))) #'(lambda (stream)(read-char-8-bit stream table)))) ((code-page-name-p external-format-name) (let ((table (cdr (assoc (external-format-id external-format) +code-page-tables+)))) #'(lambda (stream) (read-char-8-bit stream table)))) (t (case external-format-name (:utf-8 #'read-char-utf-8) (:utf-16 #'read-char-utf-16) (:utf-32 #'read-char-utf-32))))) From edi at agharta.de Wed Dec 27 21:03:28 2006 From: edi at agharta.de (Edi Weitz) Date: Wed, 27 Dec 2006 22:03:28 +0100 Subject: [flexi-streams-devel] New release 0.9.0 In-Reply-To: <4592DD6F.5020200@mail.ru> (Anton Vodonosov's message of "Wed, 27 Dec 2006 22:54:07 +0200") References: <4592DD6F.5020200@mail.ru> Message-ID: On Wed, 27 Dec 2006 22:54:07 +0200, Anton Vodonosov wrote: > Instead of creating a class for each external format in order to > optimize STREAM-READ-CHAR, we can just take all decision making code > out from STREAM-READ-CHAR to (SETF FLEXI-STREAM-EXTERNAL-FORMAT). Yes, I initially wrote something like that, but I thought it was kind of ugly and hard to maintain. Anyway, the issues with SBCL are gone now (see 0.9.1 release), so I think I'll keep it as it is for now. (I've already spent more time on this than I wanted.) Thanks for you suggestion, Edi. From attila.lendvai at gmail.com Wed Dec 27 22:39:29 2006 From: attila.lendvai at gmail.com (Attila Lendvai) Date: Wed, 27 Dec 2006 23:39:29 +0100 Subject: [flexi-streams-devel] New release 0.9.0 In-Reply-To: References: <4592DD6F.5020200@mail.ru> Message-ID: > > Instead of creating a class for each external format in order to > > optimize STREAM-READ-CHAR, we can just take all decision making code > > out from STREAM-READ-CHAR to (SETF FLEXI-STREAM-EXTERNAL-FORMAT). > > Yes, I initially wrote something like that, but I thought it was kind > of ugly and hard to maintain. Anyway, the issues with SBCL are gone a partial evaluation framework is badly missing here. oh well, even (the practical) common lisp is missing this and that... :) we experimented with similar problems using hacks like backquote stripping to have a "compiler" besides the function itself. basically generate an additional macro in a defun* which is hand annotated by backquote and comma-eval's. backquoting is stripped down in the defun version, and kept in the defmacro. with the help of a code-walker, replace calls to these defun*'s to the macro variants. it is implementation dependent, supports only a single configuration of PE, but keeps the code mostly readable and works fine in simple situations like this. traces of the hackery available (temporarily?) at: http://common-lisp.net/cgi-bin/darcsweb/darcsweb.cgi?r=cl-wdim-pmpe;a=summary (pmpe stands for poor man's partial evaluator... :) then we tried lambda's that capure and funcall other lambda's (the solution proposed in Anton's mail). works with stock cl, but the code needs to be uglyfied and inlining is not possible which can be a big loss in certain situations. and then we also tried heavy (declaim (inline ...)) annotations on sbcl and call 'compile at runtime with constant inputs to the topmost functions, but no compiler will use invariants like "this slot in this object will not change the entire time this lambda will be used", so this has limited flexibility. to make sure specialized lambda's are properly recompiled when needed, we used computed-class. but all in all, implementations are missing a way to hand-annotate dataflow-like constraints that the compiler could use to partially evaluate certain pieces of the code and recompile when a dependency is changed. (automatic discovery of this (as opposed to hand annonation) is sci-fi currently. well, for me at least... :) -- - attila "- The truth is that I've been too considerate, and so became unintentionally cruel... - I understand. - No, you don't understand! We don't speak the same language!" (Ingmar Bergman - Smultronst?llet) From vodonosov at mail.ru Wed Dec 27 22:43:31 2006 From: vodonosov at mail.ru (Anton Vodonosov) Date: Thu, 28 Dec 2006 00:43:31 +0200 Subject: [flexi-streams-devel] New release 0.9.0 In-Reply-To: References: <4592DD6F.5020200@mail.ru> Message-ID: <4592F713.1080901@mail.ru> Edi Weitz wrote: > On Wed, 27 Dec 2006 22:54:07 +0200, Anton Vodonosov wrote: > > >> Instead of creating a class for each external format in order to >> optimize STREAM-READ-CHAR, we can just take all decision making code >> out from STREAM-READ-CHAR to (SETF FLEXI-STREAM-EXTERNAL-FORMAT). >> > > Yes, I initially wrote something like that, but I thought it was kind > of ugly and hard to maintain. Do you still think so? I'm ready to rerefactor it, if you are agree that above approach is better. You pointed here - http://thread.gmane.org/gmane.lisp.steel-bank.general/1400/ - that "the code has lost quite a bit of its original elegance". It's true. Now we have new entities appeared in the library "world" only for optimization. And all optimization is to make decision only once, instead of making it every time when character is read. Regards, -Anton From edi at agharta.de Wed Dec 27 23:04:39 2006 From: edi at agharta.de (Edi Weitz) Date: Thu, 28 Dec 2006 00:04:39 +0100 Subject: [flexi-streams-devel] New release 0.9.0 In-Reply-To: <4592F713.1080901@mail.ru> (Anton Vodonosov's message of "Thu, 28 Dec 2006 00:43:31 +0200") References: <4592DD6F.5020200@mail.ru> <4592F713.1080901@mail.ru> Message-ID: On Thu, 28 Dec 2006 00:43:31 +0200, Anton Vodonosov wrote: > Do you still think so? Yes. > You pointed here - > http://thread.gmane.org/gmane.lisp.steel-bank.general/1400/ - that > "the code has lost quite a bit of its original elegance". It's > true. Now we have new entities appeared in the library "world" only > for optimization. And all optimization is to make decision only > once, instead of making it every time when character is read. Yes, but IMHO you won't get the elegance back with the approach we're talking about. I'm not too happy about the many new classes I had to introduce, but factoring out code into different methods basically seems like the right thing to me. Also note that the optimization is not only to make the decisions only once but also to rely on the implementation's method dispatch which is most likely faster than pulling a closure out of a slot and funcalling it. Anyway, I'd prefer if we leave the code as it is for now. Thanks, Edi.