From eric.marsden at free.fr  Sun Sep  3 19:35:10 2006
From: eric.marsden at free.fr (Eric Marsden)
Date: Sun, 03 Sep 2006 21:35:10 +0200
Subject: [pg-devel] Non-unicode characters are present in pg.lisp
In-Reply-To: <20060831233245.GA3653@riise-data.no> (Johan Ur Riise's message
	of "Fri, 01 Sep 2006 01:32:45 +0200")
References: <20060831233245.GA3653@riise-data.no>
Message-ID: <87irk4hhg1.fsf@free.fr>

>>>>> "jr" == Johan Ur Riise <johan at riise-data.no> writes:

  jr> When I use my CLISP with no special configuration, it assumes
  jr> files are in utf-8 encoding. Since the name of Johannes Gr??dem
  jr> has #\CEDILLA and #\LATIN_CAPITAL_LETTER_A_WITH_TILDE in some
  jr> iso-8559-variant (#xc3 #xb8), CLISP will not load it.

  on my machine, Emacs reports that pg.lisp is using UTF-8 encoding
  (and I see Johannes' name correctly). I don't know whether this is
  because the cvs client does charset conversion according to $LANG,
  or whether that's how the file is encoded in the repository. I
  haven't found any information about this in the cvsbook at
  red-bean.com ; does anyone know more?
  
-- 
Eric Marsden



From travis at travislists.com  Tue Sep  5 09:44:54 2006
From: travis at travislists.com (Travis Cross)
Date: Tue, 05 Sep 2006 05:44:54 -0400
Subject: [pg-devel] 2 New pg/sbcl test cases - 2 tests fail
Message-ID: <44FD4716.2040504@travislists.com>

Hello.

I'm running pg on Linux 2.6.16 NPTL / SBCL 0.9.16 unicode against
PostgreSQL 8.1.4. As I was noticing some issues when using the pg
library (latest cvs), I added a bit of code to pg-tests.lisp to help
in tracking the issues down.  Currently, both tests that I added
fail on SBCL.

Changes to pg-test.lisp:

 * The multiprocessing test now runs on SBCL (though it currently
generates lisp errors reliably ("The value 73 is not of type BIT.")).

 * I added a prepare/bind/execute test that tries to insert text
data.  This generates a postgresql error ("invalid byte sequence for
encoding "LATIN1": 0x00, NIL.").

 * Added #+cmu before the fwrapper code in test-fib.

I've attached backtraces of the errors that I'm seeing.  I'm
currently working through these issues to see if I can track
anything down.  Any assistance would be greatly appreciated.

Cheers,

-- Travis
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: 20060905-pg-tests-bts.lisp
URL: <https://mailman.common-lisp.net/pipermail/pg-devel/attachments/20060905/441977ed/attachment.ksh>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: pg-tests.lisp.break-on-sbcl.diff
URL: <https://mailman.common-lisp.net/pipermail/pg-devel/attachments/20060905/441977ed/attachment-0001.ksh>

From ya007 at yandex.ru  Tue Sep 12 03:56:45 2006
From: ya007 at yandex.ru (Anton)
Date: Tue, 12 Sep 2006 06:56:45 +0300
Subject: [pg-devel] pg-close-statement error
Message-ID: <FBENJDGGPLPHJJCNFLMJGEIGCCAA.ya007@yandex.ru>

File v3-protocol.lisp. Following function:

	(defmethod pg-close-statement ((connection pgcon-v3) (statement-name
string))
	    (pg-close connection statement-name #\s))

contains an error - lower case 's'. This doesn't work, PostgreSQL swars
"invalid CLOSE message subtype 115". With upper case 'S' all workds fine.

Right version:

(defmethod pg-close-statement ((connection pgcon-v3) (statement-name
string))
    (pg-close connection statement-name #\S))


Also, it would be good to have sligtly more documentation about how to use
prepared statements.
I suggest following:

1) More detailed comment for pg-close-statement in the pg.lisp:

(defgeneric pg-close-statement (connection statement-name)
  (:documentation
   "Closes prepared statement specified by statement-name;
also closes all portals associated with that statement. (see pg-prepare,
pg-bind)"))


2) Place an example, like the following one, into README or into separate
file with examples.

(defun delete-item (db-connection int-value string-value)
  (pg-prepare db-connection "delete-statement"
		  "DELETE FROM items WHERE int_column = $1 AND string_column = $2"
		  `("int4" "varchar"))
  (unwind-protect
       (progn (pg-bind db-connection "delete-portal" "delete-statement"
		       `((:int32 ,int-value) (:string ,string-value)))

	      (pg-execute db-connection "delete-portal"))

    ;; Note, portal is closed automatically
    ;; when statement is closed.
    (pg-close-statement db-connection "select-statement")))


All code is tested.

Regards,
-Anton



From erik.enge at gmail.com  Tue Sep 12 06:44:11 2006
From: erik.enge at gmail.com (Erik Enge)
Date: Tue, 12 Sep 2006 02:44:11 -0400
Subject: [pg-devel] Escaping SQL.
Message-ID: <58f839b70609112344s6fb3b27ftc1273446b90b883c@mail.gmail.com>

Hi,

I want to make sure I escape all user input to avoid SQL injection
attacks.  It is my understanding that by using PG-BIND Postgres would
do this on my behalf, however I do not currently use PG-BIND and I
think I would like to avoid it so I don't have to specify the types of
my parameters everywhere.

I can't find any SQL escape function in the pg source code nor can I
really find any mention of anyone needing such a function.  Does this
mean that most are using PG-BIND, are rolling their own, or that I
have missed something really obvious?

I wrote a small library which interfaces with libpq's PQescapeString()
function and does all the proper escaping for me.  I guess I'm asking
if you want that as a patch (perhaps not, since it requires the .so)
or would there be interest in my sharing this as a separate library?

I create a new PG connection for every thread so using
PQescapeString() works safely however I'm willing to go the extra step
and add PQescapeStringConn() if that's interesting to anyone.

Thanks,
Erik.


From eric.marsden at free.fr  Fri Sep 15 20:50:08 2006
From: eric.marsden at free.fr (Eric Marsden)
Date: Fri, 15 Sep 2006 22:50:08 +0200
Subject: [pg-devel] pg-close-statement error
In-Reply-To: <FBENJDGGPLPHJJCNFLMJGEIGCCAA.ya007@yandex.ru> (ya007@yandex.ru's
	message of "Tue, 12 Sep 2006 06:56:45 +0300")
References: <FBENJDGGPLPHJJCNFLMJGEIGCCAA.ya007@yandex.ru>
Message-ID: <87venosvlr.fsf@free.fr>

>>>>> "ay" == Anton  <ya007 at yandex.ru> writes:

  ay> contains an error - lower case 's'. This doesn't work, PostgreSQL swars
  ay> "invalid CLOSE message subtype 115". With upper case 'S' all workds fine.

  thanks, fixed.
  
  ay> Also, it would be good to have sligtly more documentation about how to use
  ay> prepared statements.

  I have added some documentation on execution plans to the README,
  together with your example. Thanks!
  
-- 
Eric Marsden



From eric.marsden at free.fr  Fri Sep 15 20:52:51 2006
From: eric.marsden at free.fr (Eric Marsden)
Date: Fri, 15 Sep 2006 22:52:51 +0200
Subject: [pg-devel] Escaping SQL.
In-Reply-To: <58f839b70609112344s6fb3b27ftc1273446b90b883c@mail.gmail.com>
	(Erik Enge's message of "Tue, 12 Sep 2006 02:44:11 -0400")
References: <58f839b70609112344s6fb3b27ftc1273446b90b883c@mail.gmail.com>
Message-ID: <87r6ycsvh8.fsf@free.fr>

>>>>> "ee" == Erik Enge <erik.enge at gmail.com> writes:

  ee> I can't find any SQL escape function in the pg source code nor can I
  ee> really find any mention of anyone needing such a function.  Does this
  ee> mean that most are using PG-BIND, are rolling their own, or that I
  ee> have missed something really obvious?

  I wrote an incomplete escaping function for a previous incarnation
  of pg-dot-lisp, that was written for scsh. I think it would be
  useful to have one for pg-dot-lisp.

  ee> I wrote a small library which interfaces with libpq's PQescapeString()
  ee> function and does all the proper escaping for me.  I guess I'm asking
  ee> if you want that as a patch (perhaps not, since it requires the .so)
  ee> or would there be interest in my sharing this as a separate library?

  I would much prefer to keep the core of pg-dot-lisp independent of
  libpq.so. Having this available as an extra file in CVS would be
  fine, though (or it should be pretty easy to implement the escaping
  in CL ...).

-- 
Eric Marsden



From eric.marsden at free.fr  Fri Sep 15 21:14:01 2006
From: eric.marsden at free.fr (Eric Marsden)
Date: Fri, 15 Sep 2006 23:14:01 +0200
Subject: [pg-devel] 2 New pg/sbcl test cases - 2 tests fail
References: <44FD4716.2040504@travislists.com>
Message-ID: <878xkksuhy.fsf@free.fr>

>>>>> "tc" == Travis Cross <travis at travislists.com> writes:

  tc> * The multiprocessing test now runs on SBCL (though it currently
  tc> generates lisp errors reliably ("The value 73 is not of type BIT.")).

  I can reproduce this every so often with SBCL 0.9.16.32 on
  Linux/AMD64. It may be a bug in SBCL's threading code. 

  tc> * I added a prepare/bind/execute test that tries to insert text
  tc> data.  This generates a postgresql error ("invalid byte sequence for
  tc> encoding "LATIN1": 0x00, NIL.").

  I can also reproduce this, but haven't worked out what it's due to.
  This error is being raised from PostgreSQL ; it's likely that some
  of the P/B/E code is sending garbage.
  
-- 
Eric Marsden



From erik.enge at gmail.com  Fri Sep 15 21:17:27 2006
From: erik.enge at gmail.com (Erik Enge)
Date: Fri, 15 Sep 2006 17:17:27 -0400
Subject: [pg-devel] Escaping SQL.
In-Reply-To: <87r6ycsvh8.fsf@free.fr>
References: <58f839b70609112344s6fb3b27ftc1273446b90b883c@mail.gmail.com>
	<87r6ycsvh8.fsf@free.fr>
Message-ID: <58f839b70609151417y157fdeacy571ee36168972058@mail.gmail.com>

On 9/15/06, Eric Marsden <eric.marsden at free.fr> wrote:
>   I would much prefer to keep the core of pg-dot-lisp independent of
>   libpq.so.  Having this available as an extra file in CVS would be
>   fine, though (or it should be pretty easy to implement the escaping
>   in CL ...).

You can write the CL one if you like but I'd personally rather use the
one in libpq.  ;-)

Note that this version only works if you access each connection is
only accessed from one thread, as explained here:
http://www.postgresql.org/docs/8.1/static/libpq-exec.html

Should probably use PQescapeStringConn() though my CFFI knowledge is
lacking and I'm not sure how to pass in a CL-created PG connection to
that function.

(cffi:define-foreign-library libpq
  (t (:default "path-to-libpq")))

(cffi:use-foreign-library libpq)

(cffi:defctype size :unsigned-int)  ; not sure size_t is unsigned-int
everywhere.

;; Note the use of kmrcl code here, replace with whatever you need.
(cffi:defcfun "PQescapeString" size
  (to :string)
  (from :string)
  (length size))

(defun sql-escape (string)
  "SQL escapes a string.

Arguments:
  string: string to escape

Returns:
  new string."
  (let* ((string (kmrcl:ensure-string string))
           (len (length string)))
    (cffi:with-foreign-string (original-string string)
      (cffi:with-foreign-pointer-as-string (buf (1+ (* 2 len)))
        (PQescapeString buf original-string len)))))


From ya007 at yandex.ru  Mon Sep 18 12:05:26 2006
From: ya007 at yandex.ru (Anton)
Date: Mon, 18 Sep 2006 15:05:26 +0300
Subject: [pg-devel] pg-bind error when working with multibyte charset
	strings; implementation-name-for-encoding improvement
Message-ID: <FBENJDGGPLPHJJCNFLMJMEJLCCAA.ya007@yandex.ru>

Hi

There is an error in pg-bind. When calculation length of a :string parameter

	(push `(:int32 ,(1+ (length value))) data)

should be changed to:

	(push `(:int32 ,(1+ (length (convert-string-to-bytes value
(pg-client-encoding connection))))) data)

Also, in the implementation-name-for-encoding function it's necessary to use
string-equal instead of string=.

PostgreSQL version I use - 8.1.3 under Windows - after executing "set
client_encoding to UTF8" "show client_encoding" returns "utf8" (lowercase).

Regards,
-Anton



From eric.marsden at free.fr  Mon Sep 18 21:34:25 2006
From: eric.marsden at free.fr (Eric Marsden)
Date: Mon, 18 Sep 2006 23:34:25 +0200
Subject: [pg-devel] pg-bind error when working with multibyte charset
	strings; implementation-name-for-encoding improvement
In-Reply-To: <FBENJDGGPLPHJJCNFLMJMEJLCCAA.ya007@yandex.ru> (ya007@yandex.ru's
	message of "Mon, 18 Sep 2006 15:05:26 +0300")
References: <FBENJDGGPLPHJJCNFLMJMEJLCCAA.ya007@yandex.ru>
Message-ID: <87bqpcdfku.fsf@free.fr>

>>>>> "ay" == Anton  <ya007 at yandex.ru> writes:

  ay> There is an error in pg-bind. When calculation length of a :string parameter

  ay> Also, in the implementation-name-for-encoding function it's necessary to use
  ay> string-equal instead of string=.

  thanks, I've fixed both of these.
  
-- 
Eric Marsden



From eric.marsden at free.fr  Mon Sep 18 21:40:07 2006
From: eric.marsden at free.fr (Eric Marsden)
Date: Mon, 18 Sep 2006 23:40:07 +0200
Subject: [pg-devel] 2 New pg/sbcl test cases - 2 tests fail
In-Reply-To: <878xkksuhy.fsf@free.fr> (Eric Marsden's message of "Fri, 15 Sep
	2006 23:14:01 +0200")
References: <44FD4716.2040504@travislists.com> <878xkksuhy.fsf@free.fr>
Message-ID: <87r6y8c0qw.fsf@free.fr>

>>>>> "ecm" == Eric Marsden <eric.marsden at free.fr> writes:
>>>>> "tc" == Travis Cross <travis at travislists.com> writes:

  tc> * I added a prepare/bind/execute test that tries to insert text
  tc> data.  This generates a postgresql error ("invalid byte sequence for
  tc> encoding "LATIN1": 0x00, NIL.").

  ecm> I can also reproduce this, but haven't worked out what it's due to.
  ecm> This error is being raised from PostgreSQL ; it's likely that some
  ecm> of the P/B/E code is sending garbage.

  I believe that I've fixed this bug. The problem was that text
  arguments to a prepared statement should be sent without a trailing
  NUL octet (unlike most other areas of the network protocol).
  
-- 
Eric Marsden



From johan at riise-data.no  Tue Sep 19 00:43:45 2006
From: johan at riise-data.no (Johan Ur Riise)
Date: Tue, 19 Sep 2006 02:43:45 +0200
Subject: [pg-devel] max message length unnecessary?
Message-ID: <20060919004345.GA24974@riise-data.no>

The limit seems to be unnecessary:

jur at lark:/usr/local/lib/common-lisp/systems/pg-cvs$ diff -u v3-protocol.lisp.orig v3-protocol.lisp
--- v3-protocol.lisp.orig       2006-09-19 03:02:13.000000000 +0200
+++ v3-protocol.lisp    2006-09-19 03:04:14.000000000 +0200
@@ -644,9 +644,6 @@
 on the database to which we are connected via CONNECTION. Return
 a result structure which can be decoded using `pg-result'."
   (let ((sql (apply #'concatenate 'simple-string args)))
-    (when (> (length sql) +MAX_MESSAGE_LEN+)
-      (error "SQL statement too long: ~A" sql))
-
     (send-packet connection #\Q `((:cstring ,sql)))
     (%flush connection)
     (do-followup-query connection)))
jur at lark:/usr/local/lib/common-lisp/systems/pg-cvs$

The limit was in effect only under a transaction.

I just tried with 8MB and it works. 


-- 
Hilsen
Johan Ur Riise


From eric.marsden at free.fr  Tue Sep 19 06:58:03 2006
From: eric.marsden at free.fr (Eric Marsden)
Date: Tue, 19 Sep 2006 08:58:03 +0200
Subject: [pg-devel] max message length unnecessary?
In-Reply-To: <20060919004345.GA24974@riise-data.no> (Johan Ur Riise's message
	of "Tue, 19 Sep 2006 02:43:45 +0200")
References: <20060919004345.GA24974@riise-data.no>
Message-ID: <87r6y8737o.fsf@free.fr>

>>>>> "jur" == Johan Ur Riise <johan at riise-data.no> writes:

  jur> The limit seems to be unnecessary:

  thanks, committed.

-- 
Eric Marsden



From eric.marsden at free.fr  Sat Sep 23 12:28:42 2006
From: eric.marsden at free.fr (Eric Marsden)
Date: Sat, 23 Sep 2006 14:28:42 +0200
Subject: [pg-devel] Socket path
In-Reply-To: <444EB492.7000102@crosswirecorp.com> (Travis Cross's message of
	"Tue, 25 Apr 2006 19:45:22 -0400")
References: <20050206182200.GD7904@parsec.no-spoon.de>
	<20060424185246.GV1818@parsec.no-spoon.de>
	<444EB492.7000102@crosswirecorp.com>
Message-ID: <874puyrclh.fsf_-_@free.fr>

>>>>> "tc" == Travis Cross <travis at crosswirecorp.com> writes:

  tc> The idea of passing in the path to the socket using the host parameter
  tc> actually seems like a reasonable solution to me, if the behavior were
  tc> consistent between implementations.

  I have (finally!) implemented this; see the following text that I
  have added to the NEWS file.


  - on CL implementations that support Unix sockets, the HOST argument
   to PG-CONNECT may designate the directory containing the local
   PostgreSQL unix socket (often "/var/run/postgresql/"). The HOST
   argument is assumed to designate a local directory rather than a
   hostname when its first character is #\/. You may need to modify
   authentication options in the PostgreSQL configuration file
   pg_hba.conf to allow connections over a unix-domain socket where
   the databse username is not equal to your ident tokens. This is an
   incompatible change to previous support for unix-domain sockets
   with CMUCL (previously a HOST of NIL told pg-dot-lisp to connect
   to a unix-domain socket whose name was hardwired into the library).
   This support currently exists for SBCL, CMUCL and OpenMCL. 

-- 
Eric Marsden



From eric.marsden at free.fr  Sun Sep 24 15:20:18 2006
From: eric.marsden at free.fr (Eric Marsden)
Date: Sun, 24 Sep 2006 17:20:18 +0200
Subject: [pg-devel] additional testing code
Message-ID: <87u02x9tql.fsf@free.fr>

Hi,

I have committed a fair amount of additional testing code to CVS. It
covers areas such as string handling, array support, bit-table
support, handling of various types of errors that can be signaled by
PostgreSQL, a simple test for transaction support, and testing for
prepared statements that use text data.

I have tested these with PostgreSQL 8.1.4 with SBCL, OpenMCL, ABCL,
CLISP. Please report any problems that you encounter!

-- 
Eric Marsden



From travis at travislists.com  Mon Sep 25 08:27:15 2006
From: travis at travislists.com (Travis Cross)
Date: Mon, 25 Sep 2006 04:27:15 -0400
Subject: [pg-devel] Socket path
In-Reply-To: <874puyrclh.fsf_-_@free.fr>
References: <20050206182200.GD7904@parsec.no-spoon.de>	<20060424185246.GV1818@parsec.no-spoon.de>	<444EB492.7000102@crosswirecorp.com>
	<874puyrclh.fsf_-_@free.fr>
Message-ID: <451792E3.7080901@travislists.com>

Eric Marsden wrote:
>>>>>> "tc" == Travis Cross <travis at crosswirecorp.com> writes:
> 
>   tc> The idea of passing in the path to the socket using the host parameter
>   tc> actually seems like a reasonable solution to me, if the behavior were
>   tc> consistent between implementations.
> 
>   I have (finally!) implemented this; see the following text that I
>   have added to the NEWS file.

Fantastic :)  Thanks for everything you do to maintain this (very
critical) library.

Cheers,

-- Travis