From piranha at thoughtcrime.us Tue Jul 24 08:46:22 2007 From: piranha at thoughtcrime.us (J.P. Larocque) Date: Tue, 24 Jul 2007 01:46:22 -0700 Subject: [postmodern-devel] Can't prepare query with :NULL Message-ID: <1185266782.13247.18.camel@synthetic-forms.thoughtcrime.us> Hi, I can't use the value :NULL in a Postmodern prepared query: > (postmodern:query "select cast ($1 as int)" :null :single) debugger invoked on a CL-POSTGRES:DATABASE-ERROR in thread #: Database error 08P01: bind message supplies 0 parameters, but prepared statement "" requires 1 Query: select cast ($1 as int) (The cast was given so that the type of the expected value would be known.) If I understand PostgreSQL and Postmodern correctly, this should return the Lisp value NIL. I have no similar problem with SQL and SQL-COMPILE, since these apparently don't technically do query preparation. I believe there is an error in the QUERY macro which is causing this. The loop which determines ARGS sees :NULL as a keyword symbol and assumes this means it is an output format. Instead of checking each member of ARGS/FORMAT to see if it is a keyword symbol, I would check for its membership in a list of possible valid format symbols. Alternatively, I'd take the idea of the DB-NULL type to an extreme, writing Lisp type definitions for each supported SQL type, and also write a type (e.g. DB-VALUE) that is the OR of all of these types. (It looks something similar was already done.) In the QUERY macro, only if an argument isn't of the type DB-VALUE would it be considered an output format. -- J.P. Larocque: , From marijnh at gmail.com Tue Jul 24 10:35:24 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Tue, 24 Jul 2007 12:35:24 +0200 Subject: [postmodern-devel] Can't prepare query with :NULL In-Reply-To: <1185266782.13247.18.camel@synthetic-forms.thoughtcrime.us> References: <1185266782.13247.18.camel@synthetic-forms.thoughtcrime.us> Message-ID: Good point, I guess I'll revise the query macro to be a bit more careful about recognizing output-format keywords. For now, you should be able to just use 'null instead. Regards, Marijn From ryszard.szopa at gmail.com Tue Jul 24 11:59:07 2007 From: ryszard.szopa at gmail.com (Ryszard Szopa) Date: Tue, 24 Jul 2007 13:59:07 +0200 Subject: [postmodern-devel] Can't prepare query with :NULL In-Reply-To: References: <1185266782.13247.18.camel@synthetic-forms.thoughtcrime.us> Message-ID: <19e19e410707240459g48367ce0h24cec72dd65a88b1@mail.gmail.com> On 7/24/07, Marijn Haverbeke wrote: > Good point, I guess I'll revise the query macro to be a bit more > careful about recognizing output-format keywords. For now, you should > be able to just use 'null instead. 'null also doesn't seem to work. (postmodern:query (format nil "select cast (~A as int)" 'null) :single) does work, though it's rather ugly. Bests, -- Richard -- http://szopa.tasak.gda.pl/ From marijnh at gmail.com Tue Jul 24 18:14:34 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Tue, 24 Jul 2007 20:14:34 +0200 Subject: [postmodern-devel] Can't prepare query with :NULL In-Reply-To: <19e19e410707240459g48367ce0h24cec72dd65a88b1@mail.gmail.com> References: <1185266782.13247.18.camel@synthetic-forms.thoughtcrime.us> <19e19e410707240459g48367ce0h24cec72dd65a88b1@mail.gmail.com> Message-ID: Ah, yes, this is a bug. I'll apply some kind of patch tomorrow. Cheers, Marijn From marijnh at gmail.com Wed Jul 25 11:17:58 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 25 Jul 2007 13:17:58 +0200 Subject: [postmodern-devel] Can't prepare query with :NULL In-Reply-To: <1185266782.13247.18.camel@synthetic-forms.thoughtcrime.us> References: <1185266782.13247.18.camel@synthetic-forms.thoughtcrime.us> Message-ID: Hi again, I ended up just fixing 'query' to be more disciminative about the things it recognizes as format-specifiers, which should solve your problem. While I was at it, I made a new release (1.02), because there were already a few other bugfixes and new features waiting. Cheers, Marijn From ryszard.szopa at gmail.com Fri Jul 27 13:02:45 2007 From: ryszard.szopa at gmail.com (Ryszard Szopa) Date: Fri, 27 Jul 2007 15:02:45 +0200 Subject: [postmodern-devel] bug in query (or deeper) Message-ID: <19e19e410707270602y32ed5384n6f40ffc79963e36a@mail.gmail.com> If you do something like this: > (postmodern::query '(:select 1)) The value (:SELECT 1) is not of type STRING. [Condition of type TYPE-ERROR] Restarts: 0: [ABORT] Return to SLIME's top level. 1: [TERMINATE-THREAD] Terminate this thread (#) 0 the slime-repl buffer starts waiting for something (probably some response from the database). When you hit C-c, and select abort, it stops. Afterwards, however, the database connection is lost: > (postmodern::query (:select 1)) Database error: Connection to database server lost. [Condition of type DATABASE-CONNECTION-LOST] Restarts: 0: [RECONNECT] Try to reconnect. 1: [ABORT] Return to SLIME's top level. 2: [TERMINATE-THREAD] Terminate this thread (#) The problem lies probably somewhere deeper than in query. If I have some time, I'll try to investigate it. Cheers, -- Richard -- http://szopa.tasak.gda.pl/ From piranha at thoughtcrime.us Sat Jul 28 06:08:55 2007 From: piranha at thoughtcrime.us (J.P. Larocque) Date: Fri, 27 Jul 2007 23:08:55 -0700 Subject: [postmodern-devel] Can't nest QUERY within prepared-statement QUERY Message-ID: <1185602935.13247.73.camel@synthetic-forms.thoughtcrime.us> Hi again, Thanks for the fix and the new release, Marijn. I think I found another problem. When I try to query with a prepared statement with the QUERY macro, and one of the arguments to the statement is the result of another QUERY macro (whether directly or from the result of a function), strange things crop up. In one instance, the inner QUERY was returning nil where it would return the expected integer otherwise. Here's an isolated case: (postmodern:query "select cast ($1 as int)" (let ((foo (postmodern:query "select 12" :single))) (format t "result: ~S~%" foo) foo)) This gives me the expected message "result: 12", but also signals a condition, "Database error: Incorrect number of parameters given for prepared statement ." Below is the complete macroexpansion of the above form (without LET and FORMAT). (PROGN (CL-POSTGRES:PREPARE-QUERY POSTMODERN:*DATABASE* "" "select cast ($1 as int)") (CL-POSTGRES:EXEC-PREPARED POSTMODERN:*DATABASE* "" (MAPCAR 'S-SQL:SQL-IZE (LIST (CAR (CL-POSTGRES:EXEC-QUERY POSTMODERN:*DATABASE* "select 12" 'POSTMODERN::COLUMN-ROW-READER)))) 'CL-POSTGRES:LIST-ROW-READER)) The sequence of evaluation is PREPARE-QUERY, then EXEC-QUERY (for the inner "select 12"), then EXEC-PREPARED. I guessed that doing things with the database in-between PREPARE-QUERY and EXEC-PREPARED might be the problem, so I modified QUERY to evaluate prepared-statement arguments before PREPARE-QUERY. This allows the inner query to complete before the outer one actually starts. diff -urNx '*~' postmodern-1.02/postmodern/query.lisp postmodern-1.02-modified/postmodern/query.lisp --- postmodern-1.02/postmodern/query.lisp 2007-07-25 04:10:42.000000000 -0700 +++ postmodern-1.02-modified/postmodern/query.lisp 2007-07-27 23:03:16.000000000 -0700 @@ -55,9 +55,10 @@ :else :collect arg))) (destructuring-bind (reader single-row) (cdr (assoc format *result-styles*)) (let ((base (if args - `(progn - (prepare-query *database* "" ,(real-query query)) - (exec-prepared *database* "" (mapcar 'sql-ize (list , at args)) ',reader)) + (let ((arg-values (gensym))) + `(let ((,arg-values (mapcar 'sql-ize (list , at args)))) + (prepare-query *database* "" ,(real-query query)) + (exec-prepared *database* "" ,arg-values ',reader))) `(exec-query *database* ,(real-query query) ',reader)))) (if single-row `(multiple-value-call 'car-of-first-value ,base) Now my first example works, as does the result of one prepared-statement QUERY's being indirectly nested in another prepared-statement QUERY. -- J.P. Larocque: , From marijnh at gmail.com Sat Jul 28 14:38:41 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sat, 28 Jul 2007 16:38:41 +0200 Subject: [postmodern-devel] bug in query (or deeper) In-Reply-To: <19e19e410707270602y32ed5384n6f40ffc79963e36a@mail.gmail.com> References: <19e19e410707270602y32ed5384n6f40ffc79963e36a@mail.gmail.com> Message-ID: Ryszard, Thanks for the report. That is rather clumsy indeed -- what happens is that the libary sends the start of a message to the server, and then raises an error when it tries to send the list as a string. After such an error, it will try to re-synchronize the connection, so that it can be used again without reconnecting, but a half-sent message puts the connection in such a broken state that it can not be re-synchronized -- hence the hanging you saw, the server wasn't responding anymore. I pushed a simple patch that uses check-type to make sure queries (and query names) passed to the cl-postgres functions are string values. Cheers, Marijn From marijnh at gmail.com Sat Jul 28 14:50:38 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sat, 28 Jul 2007 16:50:38 +0200 Subject: [postmodern-devel] Mainainer absence Message-ID: Hello Postmodern-users, Starting somewhere at the end of next week, I will be travelling. How long this is going to take is not clear yet, but it will mean that I will be less actively maintaining the library. I will read my e-mail now and then, and applying a patch sent to me should be possible whenever I come across an internet cafe that lets me use SSH, but I certainly won't be writing any patches or properly testing things sent to me. If anyone feels up to it, I wouldn't mind giving someone else commit rights to the repository. (I will, of course, still be hanging over your shoulder trying to enforce my extremist ideas about clean interfaces and up-to-date documentation.) Another option is that somebody sets up a 'secondary' repository, advertises it on this list, and keeps it up to date with whatever bug-fixes and features people come up with. Cheers, Marijn From marijnh at gmail.com Sat Jul 28 16:28:13 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sat, 28 Jul 2007 18:28:13 +0200 Subject: [postmodern-devel] Can't nest QUERY within prepared-statement QUERY In-Reply-To: <1185602935.13247.73.camel@synthetic-forms.thoughtcrime.us> References: <1185602935.13247.73.camel@synthetic-forms.thoughtcrime.us> Message-ID: Hi, Good catch. I applied a fix, more or less your patch but without the (unneccesary) gensym. Cheers, Marijn From piranha at thoughtcrime.us Sun Jul 29 19:17:23 2007 From: piranha at thoughtcrime.us (J.P. Larocque) Date: Sun, 29 Jul 2007 12:17:23 -0700 Subject: [postmodern-devel] Micro-optimizing SQL-ESCAPE-STRING, ESCAPE-BYTES In-Reply-To: References: <1185602935.13247.73.camel@synthetic-forms.thoughtcrime.us> Message-ID: <1185736643.13247.95.camel@synthetic-forms.thoughtcrime.us> On Sat, 2007-07-28 at 18:28 +0200, Marijn Haverbeke wrote: > Good catch. I applied a fix, more or less your patch but without the > (unneccesary) gensym. Right, cargo-cult habits at work. =) I'm doing some stuff involving binary data (geometry information with PostGIS), and I've found encoding binary data with SQL-COMPILE is very slow. For instance, 91 seconds for a 50KiB (VECTOR (UNSIGNED-BYTE 8)). The slowness is in S-SQL:ESCAPE-BYTES and S-SQL:SQL-ESCAPE-STRING. It turns out SBCL's WITH-OUTPUT-TO-STRING is quadratic in time, unless (as Juho Snellman pointed out on #lisp) you bind *PRINT-PRETTY* to NIL. These functions are linear on both CLISP and OpenMCL. Before he mentioned that, I spent some time micro-optimizing these two functions. They're ugly--not entirely unlike hairier C code. You could say I ended up implementing a very specific, low-level, non-general form of the functionality of WITH-OUTPUT-TO-STRING. SQL-COMPILE on the same expression now takes 0.3 seconds, though, and they are about four times faster than simply making that binding. This could be especially important for more heavy-weight users of binary data in PostgreSQL--say, back-ends for object databases. (My own requirement for large binary data is temporary.) If you're interested, I've attached a file with these functions and some comparison and testing functions (even uglier). If nothing else, could you bind *PRINT-PRETTY* to NIL for the above two functions? Thanks, -- J.P. Larocque: , -------------- next part -------------- ;;; These are the efficient, albeit hairy and micro-optimized versions ;;; of ESCAPE-BYTES and SQL-ESCAPE-STRING. ;;; These functions were tested on the following implementations. ;;; Rates are from the measurement of (SQL-ESCAPE-STRING (ESCAPE-BYTES ;;; v)), without OPTIMIZE declarations, where v is a vector of ;;; 10,000,000 random octets, on a 2GHz Athlon64. ;;; SBCL 0.9.16 x86_64: 860,807 octets/sec ;;; OpenMCL 1.1-pre-070512 LinuxX8664: 627,077 octets/sec ;;; CLISP 2.41 x86_64: 18,243 octets/sec ;;; (To its credit, CLISP used under 100MiB of RAM, where the ;;; others used over ~200MiB.) (defmacro defencoder (name &key get-length set-string documentation (prefix "") (suffix "") (element-type t) (character-type 'character)) "Defines a function of the given NAME which encodes an input vector to a string. The resulting string always starts with the contents of PREFIX and always ends with the contents of SUFFIX. Each element of the defined function's input is mapped to zero or more characters by the given GET-LENGTH and SET-STRING functions. GET-LENGTH takes one argument, an element, and returns the number of characters required to encode it. SET-STRING takes three arguments--an element, a string, and a position--which is required to write the encoded form of the element to the given string at the given position, and then return the number of characters written. This unfortunately low-level interface was selected to avoid consing and slow implementations of WITH-OUTPUT-TO-STRING. The type of the elements of the input vector may be specified with ELEMENT-TYPE. The resulting string will be a subtype of `(VECTOR ,CHARACTER-TYPE). When CHARACTER-TYPE is supplied, SET-STRING must write only characters of that type." (when (or (null get-length) (null set-string)) (error "Must supply :GET-LENGTH and :SET-STRING.")) `(let ((get-length-fn ,get-length) (set-string-fn ,set-string) (prefix ,prefix) (suffix ,suffix)) (defun ,name (input-vec) ,@(if (null documentation) '() `(,documentation)) (let* ((output-length (+ (loop :for element :of-type ,element-type :across input-vec :sum (funcall get-length-fn element)) (length prefix) (length suffix))) (output (make-array output-length :element-type ',character-type ;; #\Space for lack of something better. :initial-element #\Space)) (output-pos 0)) (flet ((add-string (s) (setf (subseq output output-pos (+ output-pos (length s))) s) (incf output-pos (length s)))) (add-string prefix) (loop :for element :of-type ,element-type :across input-vec :for written-count := (funcall set-string-fn element output output-pos) :do (incf output-pos written-count)) (add-string suffix)) output)))) (defencoder escape-bytes :documentation "Escape an array of octets in PostgreSQL's horribly inefficient textual format for binary data." ;; STANDARD-CHAR contains backslash, digits, and all the ASCII ;; characters we pass unescaped. :element-type (unsigned-byte 8) :character-type standard-char :get-length (lambda (byte) (if (or (< byte 32) (> byte 126) (= byte 39) (= byte 92)) 4 ; "\ooo" 1)) :set-string (lambda (byte s start) (cond ((or (< byte 32) (> byte 126) (= byte 39) (= byte 92)) ;; Write "\ooo". (let ((place64 (floor byte 64)) (place8 (mod (floor byte 8) 8)) (place1 (mod byte 8))) (setf (elt s (+ start 0)) #\\) (setf (elt s (+ start 1)) (digit-char place64 8)) (setf (elt s (+ start 2)) (digit-char place8 8)) (setf (elt s (+ start 3)) (digit-char place1 8)) 4)) (t (setf (elt s start) (code-char byte)) 1)))) (defencoder sql-escape-string :documentation "Escape string data so it can be used in a query." :prefix "'" :suffix "'" :element-type character :get-length (lambda (char) (if (member char '(#\' #\\) :test #'char=) 2 ; "''" or "\\\\", respectively. 1)) :set-string (lambda (char s start) (case char (#\' ;; Write "''". (setf (elt s start) #\') (setf (elt s (1+ start)) #\') 2) (#\\ ;; Write "\\\\". Turn off postgres' backslash ;; behaviour to prevent unexpected strangeness. (setf (elt s start) #\\) (setf (elt s (1+ start)) #\\) 2) (t (setf (elt s start) char) 1)))) ;;; These versions--the originals--take quadratic time on SBCL, ;;; because of WITH-OUTPUT-TO-STRING. That's SBCL's fault; with ;;; OpenMCL, the time is linear. (defun escape-bytes-orig (bytes) "Escape an array of octets in PostgreSQL's horribly inefficient textual format for binary data." (with-output-to-string (out) (loop :for byte :of-type fixnum :across bytes :do (if (or (< byte 32) (> byte 126) (= byte 39) (= byte 92)) (format out "\\~3,'0o" byte) (princ (code-char byte) out))))) (defun sql-escape-string-orig (string) "Escape string data so it can be used in a query." (with-output-to-string (out) (princ #\' out) (loop :for char :of-type character :across string :do (princ (case char (#\' "''") ;; Turn off postgres' backslash behaviour to ;; prevent unexpected strangeness. (#\\ "\\\\") (t char)) out)) (princ #\' out))) ;;; Some utility functions for comparing the above. (defun time-escape-function (escape-f input-vec) (let* ((start-real-time (get-internal-real-time)) (start-run-time (get-internal-run-time)) (test-result (time (funcall escape-f input-vec)))) (let* ((end-real-time (get-internal-real-time)) (end-run-time (get-internal-run-time)) (real-time (/ (- end-real-time start-real-time) internal-time-units-per-second)) (run-time (/ (- end-run-time start-run-time) internal-time-units-per-second)) (real-time-rate (unless (zerop real-time) (/ (length input-vec) real-time))) (run-time-rate (unless (zerop run-time) (/ (length input-vec) run-time)))) (format t "Real time: ~,1F (~,1F elt/sec)~%Run time: ~,1F (~,1F elt/sec)~%" real-time real-time-rate run-time run-time-rate) test-result))) (defun compare-escape-functions (input-size &rest escape-functions) (let ((test-vec (make-array `(,input-size) :element-type '(unsigned-byte 8) :initial-element 0)) last-result) (loop for i from 0 below input-size doing (setf (elt test-vec i) (random 256))) (dolist (escape-function escape-functions) (format t "~S:~%" escape-function) (let ((result (time-escape-function escape-function test-vec))) (unless (null last-result) (if (equalp result last-result) (format t "Results match.~%") (format t "*** Results do not match.~%"))) (terpri) (setf last-result result)))) (values)) (defvar *failed-vector* nil ;; We have this because the failing vector might be huge and too ;; cumbersome for a condition or result value to a REPL. "The last vector that failed within TEST-ESCAPE-FUNCTIONS.") (defun test-escape-functions (type initial-element key elt-upper-bound escape-f-1 escape-f-2) (labels ((test-vector (vector desc &rest arguments) (let ((result1 (funcall escape-f-1 vector)) (result2 (funcall escape-f-2 vector))) (unless (equalp result1 result2) (setf *failed-vector* vector) (error "~S and ~S return different results for: ~?" escape-f-1 escape-f-2 desc arguments))))) (test-vector (make-array 0 :element-type type) "empty vector") (loop :for len :from 1 :below 10 :do (loop :for i :from 0 :below elt-upper-bound :for elt := (funcall key i) :do (unless (null elt) (let ((vector (make-array len :element-type type :initial-element elt))) (test-vector vector "length ~D with element ~S" len elt))))) (loop :for len := (random 10) :for vector := (make-array len :element-type type :initial-element initial-element) :repeat 25 :do (loop :for i :from 0 :below len :for elt := (loop :for eltx := (funcall key (random elt-upper-bound)) :while (null eltx) :finally (return eltx)) :doing (setf (elt vector i) elt)) :do (test-vector vector "~D random element~:P" len)))) ;;; On Lisp, 5.4: Composing Functions (defun compose (&rest functions) (if (endp functions) #'identity (let ((function1 (car (last functions))) (functions (butlast functions))) (lambda (&rest args) (reduce #'funcall functions :from-end t :initial-value (apply function1 args)))))) (defun run-tests () (format t "Testing ESCAPE-BYTES{,-ORIG}.~%") (test-escape-functions '(unsigned-byte 8) 0 #'identity 256 #'escape-bytes #'escape-bytes-orig) ;; Cap at 64K to avoid going through 1M+ Unicode characters. (format t "Testing SQL-ESCAPE-STRING{,-ORIG}.~%") (test-escape-functions 'character #\Space #'code-char (min char-code-limit 65536) #'sql-escape-string #'sql-escape-string-orig) (format t "Testing the composition of SQL-ESCAPE-STRING ESCAPE-BYTES with ...-ORIG.~%") (test-escape-functions '(unsigned-byte 8) 0 #'identity 256 (compose #'sql-escape-string #'escape-bytes) (compose #'sql-escape-string-orig #'escape-bytes-orig)) (format t "No problems to report.")) From marijnh at gmail.com Sun Jul 29 19:37:31 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sun, 29 Jul 2007 21:37:31 +0200 Subject: [postmodern-devel] Micro-optimizing SQL-ESCAPE-STRING, ESCAPE-BYTES In-Reply-To: <1185736643.13247.95.camel@synthetic-forms.thoughtcrime.us> References: <1185602935.13247.73.camel@synthetic-forms.thoughtcrime.us> <1185736643.13247.95.camel@synthetic-forms.thoughtcrime.us> Message-ID: Hahah, 91 seconds is definitely bad -- postgres' binary-data escaping method is pretty stupid, and I implemented that encoder more or less as an afterthought. The performance improvement of your code is impressive... but it is indeed hideous. I'll look into this in the next few days, I'm still hoping I can find a middle ground -- something that performs decently but does not require four screenfuls of c code ;) Cheers, Marijn From marijnh at gmail.com Mon Jul 30 20:12:53 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Mon, 30 Jul 2007 22:12:53 +0200 Subject: [postmodern-devel] Micro-optimizing SQL-ESCAPE-STRING, ESCAPE-BYTES In-Reply-To: <1185736643.13247.95.camel@synthetic-forms.thoughtcrime.us> References: <1185602935.13247.73.camel@synthetic-forms.thoughtcrime.us> <1185736643.13247.95.camel@synthetic-forms.thoughtcrime.us> Message-ID: Okay, I have to take back my remarks about the code being ugly -- it is pretty cool actually, very correct and not as big as I originally thought, since most of it is benchmarking code. Still, using textual queries for sending big amounts of binary data just isn't a very good idea, so I'm reluctant to add complex optimizations for that. On the bright side, I was able to reach results almost as good as your code (on SBCL at least, didn't test other implementations) by just turning off *PRINT-PRETTY* and not using FORMAT to create the octal values. A patch for that has been pushed. cl-postgres has very simple support for sending binary data unescaped -- see the 'blob' test in its test suite for that. But the postmodern wrapper doesn't make use of this yet -- it was added by one of Attila's patches, and not part of my original 'vision' (heh). You could look into changing things so that blobs are passed through unescaped when they are given as 'parameters' to (prepared) queries. I'm not sure how to do that in a nice way -- maybe some kind of generalized system where binary sender functions can be registered for specific types. Cheers, Marijn