From 00003b at gmail.com Thu Mar 1 19:47:18 2007 From: 00003b at gmail.com (Bart Botta) Date: Thu, 1 Mar 2007 13:47:18 -0600 Subject: [postmodern-devel] bug in cl-postgres/protocol.lisp? Message-ID: <77cb99c00703011147x3996eaa8tcff15244844c9f1e@mail.gmail.com> Using version 7.4.7 of PostgreSQL, SBCL 1.0, postmodern 0.23, the deftable/create-table example breaks due to WARNING: Postgres warning: CREATE TABLE / PRIMARY KEY will create implicit index "test_data_pkey" for table "test_data" which causes cl-postgres::send-query to drop into the debugger with Postgresql protocol error: Unexpected message received: 1 [Condition of type CL-POSTGRES::PROTOCOL-ERROR] If I read the code and postgres protocol docs correctly, this is due to the way message-case handles NoticeResponse (#\N) messages, specifically that it only reads 1 of #\E, #\N, or the expected message, so when it receives both #\N and the expected message, the caller gets out of sync. Removing (#.(char-code #\N) (get-warning socket)) from the case and adding (when (= #.(char-code #\N) ,char-name) (get-warning socket) (setf ,char-name (read-uint1 ,socket-name)) (setf ,size-name (read-uint4 ,socket-name))) before the (case ,char-name ...) in message-case fixes the problem for me, not sure if it is completely correct though, possibly should loop in case there are multiple warnings? (Also, shouldn't that (get-warning socket) and the get-error call use ,socket-name instead of socket?) -- Bart From marijnh at gmail.com Sat Mar 3 16:31:11 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sat, 3 Mar 2007 17:31:11 +0100 Subject: [postmodern-devel] bug in cl-postgres/protocol.lisp? In-Reply-To: <77cb99c00703011147x3996eaa8tcff15244844c9f1e@mail.gmail.com> References: <77cb99c00703011147x3996eaa8tcff15244844c9f1e@mail.gmail.com> Message-ID: Hello Bart, Sorry about the somewhat late response, my Internet access is limited at the moment. Thanks for the quality bug report. I pushed something that should fix this to the repo. For some reason I never had any problems with this bug, even though I do get the create-table warning all the time (the tests even get it, do they fail on your setup?). Good catch about the use of 'socket' instead of ',socket-name' too. Let me know if this patch solves your problem. Regards, Marijn From 00003b at gmail.com Tue Mar 6 17:26:00 2007 From: 00003b at gmail.com (Bart Botta) Date: Tue, 6 Mar 2007 11:26:00 -0600 Subject: [postmodern-devel] bug in cl-postgres/protocol.lisp? In-Reply-To: <77cb99c00703051301g54c9eeferedad4c435ab9b960@mail.gmail.com> References: <77cb99c00703011147x3996eaa8tcff15244844c9f1e@mail.gmail.com> <77cb99c00703051301g54c9eeferedad4c435ab9b960@mail.gmail.com> Message-ID: <77cb99c00703060926h3696497ak50d9b27a474ddff3@mail.gmail.com> (oops, forgot to send the message quoted below to the list) After looking into the time/interval errors a bit more, it looks like at least part of the problem with intervals is that the format of intervals changed with postgres 8.1 (adding the day field between ms and month), so I think I will just upgrade postgres rather than worry about any other hidden incompatibilities. -- Bart On 3/5/07, Bart Botta <00003b at gmail.com> wrote: > The new version fixes the problem. I checked the tests, and they fail > the same way with version 0.23. > > WARNING: > Postgres warning: CREATE TABLE / PRIMARY KEY will create implicit > index "test_data_pkey" for table "test_data" > WARNING: > Postgres warning: CREATE TABLE / UNIQUE will create implicit index > "test_data_c_key" for table "test_data" > ... > Did 26 checks. > Pass: 22 (84%) > Skip: 0 ( 0%) > Fail: 4 (15%) > > Failure Details: > -------------------------------- > TABLE []: > Unexpected Error: # > Postgresql protocol error: Unexpected message received: 1.. > -------------------------------- > -------------------------------- > TABLE []: > Unexpected Error: # > # is closed... > -------------------------------- > > > Version 0.23 and the new version also fail the time-types test, > > -------------------------------- > TIME-TYPES []: > (ENCODE-TIMESTAMP 2040 3 19 12 15 0 2) evaluated to # 19-03-2040T12:15:00,002>, which is not TIME= to # 01-01-2000T00:00:00>.. > -------------------------------- > -------------------------------- > TIME-TYPES []: > Unexpected Error: # > Postgresql protocol error: Unexpected message received: ^K.. > -------------------------------- > > > The cl-postgres tests lock up in the date-query test, > (postmodern:query "select '2 years -4 days'::interval") > by itself causes > > arithmetic error FLOATING-POINT-OVERFLOW signalled > [Condition of type FLOATING-POINT-OVERFLOW] > > Backtrace: > ... > 7: (SB-KERNEL:TWO-ARG-* 0.0d0 1000) > 8: (CL-POSTGRES::INTERPRET-MILLISECS 18446743728109551616) > > > -- > Bart > From marijnh at gmail.com Wed Mar 7 13:10:23 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 7 Mar 2007 14:10:23 +0100 Subject: [postmodern-devel] bug in cl-postgres/protocol.lisp? In-Reply-To: <77cb99c00703060926h3696497ak50d9b27a474ddff3@mail.gmail.com> References: <77cb99c00703011147x3996eaa8tcff15244844c9f1e@mail.gmail.com> <77cb99c00703051301g54c9eeferedad4c435ab9b960@mail.gmail.com> <77cb99c00703060926h3696497ak50d9b27a474ddff3@mail.gmail.com> Message-ID: On 3/6/07, Bart Botta <00003b at gmail.com> wrote: > After looking into the time/interval errors a bit more, it looks like > at least part of the problem with intervals is that the format of > intervals changed with postgres 8.1 (adding the day field between ms > and month), so I think I will just upgrade postgres rather than worry > about any other hidden incompatibilities. That's good to know -- I'll add something about that to the compatibility info. Did you find out in which version exactly this change was introduced? Marijn From 00003b at gmail.com Wed Mar 7 17:04:38 2007 From: 00003b at gmail.com (Bart Botta) Date: Wed, 7 Mar 2007 11:04:38 -0600 Subject: [postmodern-devel] bug in cl-postgres/protocol.lisp? In-Reply-To: References: <77cb99c00703011147x3996eaa8tcff15244844c9f1e@mail.gmail.com> <77cb99c00703051301g54c9eeferedad4c435ab9b960@mail.gmail.com> <77cb99c00703060926h3696497ak50d9b27a474ddff3@mail.gmail.com> Message-ID: <77cb99c00703070904j10a092a5s735b2fa84b7498df@mail.gmail.com> On 3/7/07, Marijn Haverbeke wrote: > That's good to know -- I'll add something about that to the > compatibility info. Did you find out in which version exactly this > change was introduced? > > Marijn Looks like it was 8.1.0. from the 8.1 release notes at http://www.postgresql.org/docs/8.1/interactive/release-8-1.html#AEN73858 Add a separate day field to type interval so a one day interval can be distinguished from a 24 hour interval (Michael Glaesemann) Days that contain a daylight saving time adjustment are not 24 hours long, but typically 23 or 25 hours. This change creates a conceptual distinction between intervals of "so many days" and intervals of "so many hours". Adding 1 day to a timestamp now gives the same local time on the next day even if a daylight saving time adjustment occurs between, whereas adding 24 hours will give a different local time when this happens. For example, under US DST rules: '2005-04-03 00:00:00-05' + '1 day' = '2005-04-04 00:00:00-04' '2005-04-03 00:00:00-05' + '24 hours' = '2005-04-04 01:00:00-04' -- Bart From jcorneli at planetmath.org Wed Mar 7 23:12:45 2007 From: jcorneli at planetmath.org (Joe Corneli) Date: Wed, 7 Mar 2007 18:12:45 -0500 (EST) Subject: [postmodern-devel] fragile connection when creating new table? Message-ID: <20070307231245.0A37B81AD@planetmath.cc.vt.edu> Hi, I'm using postmodern under OpenMCL. Connected to an empty database, I tried defining and creating a new table. Doing this directly from the psql prompt gave a bunch of messages, but doing it from within Lisp caused serious trouble. Compare: 1. jcorneli=# create table strs (strid SERIAL PRIMARY KEY, string TEXT NOT NULL UNIQUE); NOTICE: CREATE TABLE will create implicit sequence "strs_strid_seq" for serial column "strs.strid" NOTICE: CREATE TABLE will create implicit sequence "strs_strid_seq" for serial column "strs.strid" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "strs_pkey" for table "strs" NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "strs_pkey" for table "strs" NOTICE: CREATE TABLE / UNIQUE will create implicit index "strs_string_key" for table "strs" NOTICE: CREATE TABLE / UNIQUE will create implicit index "strs_string_key" for table "strs" CREATE TABLE 2. ? (deftable strs () ((str_id :type serial :primary-key t) (string :type text :unique t)) (:indices str_id)) ? (create-table 'strs) ; Warning: Postgres warning: CREATE TABLE will create implicit sequence "strs_str_id_seq" for serial column "strs.str_id" ; While executing: CL-POSTGRES::GET-WARNING > Error in process listener(1): Postgresql protocol error: Unexpected message received: 1 > While executing: CL-POSTGRES::SEND-QUERY > Type :POP to abort. Type :? for other options. 1 > :POP > Error in process listener(1): # is closed > While executing: CCL::STREAM-IOBLOCK > Type :POP to abort. Type :? for other options. 1 > :POP This looks like a bug to me. From jcorneli at planetmath.org Thu Mar 8 00:12:45 2007 From: jcorneli at planetmath.org (Joe Corneli) Date: Wed, 7 Mar 2007 19:12:45 -0500 (EST) Subject: [postmodern-devel] fragile connection under OpenMCL in general? In-Reply-To: <20070307231245.0A37B81AD@planetmath.cc.vt.edu> (message from Joe Corneli on Wed, 7 Mar 2007 18:12:45 -0500 (EST)) References: <20070307231245.0A37B81AD@planetmath.cc.vt.edu> Message-ID: <20070308001245.AB69281C8@planetmath.cc.vt.edu> I have a table strs, now, that looks like this: select * from strs; strid | string -------+-------- 1 | car 2 | has 3 | wheels 4 | bike Attempting to run this under postmodern gives me an error: ? (execute (:insert-into 'strs :set 'string "predicate")) > Error in process listener(1): Database error 08P01: invalid message format > Query: INSERT INTO strs (string) VALUES ('predicate') > While executing: CL-POSTGRES::GET-ERROR > Type :POP to abort. Type :? for other options. 1 > :POP > Error in process listener(1): Database error 08P01: invalid frontend message type 0 > While executing: CL-POSTGRES::GET-ERROR > Type :POP to abort. Type :? for other options. 1 > :POP From marijnh at gmail.com Fri Mar 9 15:05:46 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Fri, 9 Mar 2007 16:05:46 +0100 Subject: [postmodern-devel] fragile connection when creating new table? In-Reply-To: <20070307231245.0A37B81AD@planetmath.cc.vt.edu> References: <20070307231245.0A37B81AD@planetmath.cc.vt.edu> Message-ID: Hello Joe, It looks like this is related to the issue reported by Bart Botta this week. There is a fix for that in the repository, but no new release has been made yet. Could you try fetching the latest files with darcs to see if this problem (and the in your other mail) gets solved by that? Regards, Marijn From henrik at evahjelte.com Fri Mar 9 17:42:44 2007 From: henrik at evahjelte.com (Henrik Hjelte) Date: Fri, 09 Mar 2007 18:42:44 +0100 Subject: [postmodern-devel] A new elephant backend Message-ID: <1173462164.7459.126.camel@trinidad> First, thanks for postmodern, it's a great library! I just used it to make a new backend for the elephant object-store, db-postmodern. Cross-posting from elephant-devel: It uses postgresql for storage, and the postmodern lisp socket interface instead of clsql. The implementation is actually quite different from the clsql backend. db-postmodern uses prepared queries and stored procedures for performance reasons. The db-postmodern cursor is based on a sql cursor. Each btree has its own database table. db-postmodern is slower than the berkeley db backend, I guess about four or five times. It now passes all tests and the migration test, but should of course be regarded as an alpha. It is only tested on sbcl, but should work on all platforms that postmodern supports. About postmodern: http://common-lisp.net/project/postmodern/ db-postmodern mainly uses the lower level cl-postgres driver in the postmodern project. During development I used a darcs mirror of elephant, and I have temporarily put it at common-lisp.net. This darcs branch is intended for people wanting to try db-postmodern, and it is not an automatic mirror of the cvs elephant. Download: To get elephant containing db-postmodern, darcs get http://common-lisp.net/project/grand-prix/darcs/elephant Best wishes, Henrik Hjelte From jcorneli at planetmath.org Fri Mar 9 20:34:38 2007 From: jcorneli at planetmath.org (Joe Corneli) Date: Fri, 9 Mar 2007 15:34:38 -0500 (EST) Subject: [postmodern-devel] fragile connection when creating new table? In-Reply-To: (marijnh@gmail.com) References: <20070307231245.0A37B81AD@planetmath.cc.vt.edu> Message-ID: <20070309203438.D036B81ED@planetmath.cc.vt.edu> That seems to have taken care of the problem, thanks! From vamlists at gmail.com Sun Mar 25 08:00:14 2007 From: vamlists at gmail.com (Vamsee Kanakala) Date: Sun, 25 Mar 2007 13:30:14 +0530 Subject: [postmodern-devel] Bug in :limit, patch for s-sql.lisp Message-ID: <46062C0E.1040304@gmail.com> Hi, Looks like :limit works only partly - according to the documentation, "It limits the number of results to the amount given as the second argument, or when three arguments are given it returns the results between the two given numbers". However, if I give the 3rd value, say like this: (query (:limit (:order-by (:select 'id 'name :from 'cities) 'name) 10 20)) I get an error like this: Database error 42601: LIMIT #,# syntax is not supported Use separate LIMIT and OFFSET clauses. Query: (((SELECT id, name FROM cities) ORDER BY name) LIMIT 10, 20) Looks like :offset is not implemented yet. I have attached a patch to fix these issues, kindly let me know the preferred way of sending patches (I used darcs diff --unified) so that I can help out where I can. Thanks much, Vamsee. -------------- next part -------------- A non-text attachment was scrubbed... Name: s-sql.lisp.patch Type: text/x-patch Size: 801 bytes Desc: not available URL: From vamlists at gmail.com Tue Mar 27 09:38:30 2007 From: vamlists at gmail.com (Vamsee Kanakala) Date: Tue, 27 Mar 2007 15:08:30 +0530 Subject: [postmodern-devel] Help with query replacement Message-ID: <4608E616.1040507@gmail.com> Hi, I'm trying add paging support to my web pages, so it's important that I know the number of results a query will generate, so that I can create the appropriate paging code. I'm trying to change something like this: (query-dao 'city (:select 'id 'name :from 'cities)) to (query (:select (:raw "count(*)") :from 'cities)) So that I can get the number of results first and then run the actual query with limit and offset. I thought it best not to change postmodern code itself, to avoid hassles when a new version comes out. However, parsing the above query with a recursive function to replace 'id and 'name with (:raw "count(*)") is proving to be beyond my lisp-fu. Any easy way to achieve this? My terrible hack to do this: (defun query-subst (list) (if (null list) (list) (if (atom (car list)) (progn (cond ((eql (car list) :select) (setf skip t)) ((eql (car list) :from) (setf skip nil))) (if (or (null skip) (eql (car list) :select)) (cons (car list) (query-subst (cdr list))) (if (null count-set) (progn (setf count-set t) (list :raw "count(*)")) (query-subst (cddr list))))) (cons (query-subst (car list)) (query-subst (cdr list)))))) But this doesn't really do my job, because my query ends up like this: (query (:select (:raw "count(*)") nil :from 'cities)) Any way to avoid those extra nils? Probably c.l.l is a better place to post this, but wanted to know if there's a less complicated way to do this. Thanks, Vamsee. From vamlists at gmail.com Tue Mar 27 17:34:11 2007 From: vamlists at gmail.com (Vamsee Kanakala) Date: Tue, 27 Mar 2007 23:04:11 +0530 Subject: [postmodern-devel] Help with query replacement In-Reply-To: References: <4608E616.1040507@gmail.com> Message-ID: <46095593.2060202@gmail.com> Marijn Haverbeke wrote: > > Unfortunately I don't quite understand what you are trying to do, but > it seems that in the simple case something like this (untested) might > work... > Sorry, I forgot that using the word paging without the context can be confusing. By paging, I meant paging in the web pages - if I have 100 results to a query, I would like to show that to the user in chunks of 20 rows over 5 pages, to avoid huge requests and make the user's life easier. To avoid putting this 100-row result in the session and then paging over it (can be quite inefficient if there are lots of results), I want to retrieve the count of results, and then retrieve small piece of them according to page numbers with :offset and :limit. So something like this: (query (:select 'id 'name :from 'cities)) Would turn into: (query-dao 'city (:limit (:offset (:order-by (:select 'id 'name :from 'cities) 'name) offset) limit)) But to calculate how many pages I have to create links to, I need to have the number of results before the query actually runs. So, number of results divided by limit should give me the number of pages this result set would run into. But to avoid writing a similar query twice, I'm trying to change the first query into this (the aim is to write a macro which does this automatically), to get the number of rows: (query (:select :raw("count(*)") :from 'cities) :single) Also, the element :count was not added to postmodern - I have checked the source too. So, I have to use :raw for now - please correct me if I'm wrong. > (defun query-subst (query) > (let ((from (position :from query))) > `(:select (:count '*) ,@(nthcdr from query))) > Perhaps you can already see that this does not help - position cannot look into trees (works only with sequences) and the select statement can be arbitrarily long. I think my problem can be simply put this way: I need to replace everything between :select and :from with a single (:count *) or (:raw "count(*)"). Perhaps I'm being finicky - I'm just looking to cut some extra lines of code and move the entire paging-related code into one macro. Thanks much, Vamsee. From marijnh at gmail.com Sat Mar 31 10:19:38 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sat, 31 Mar 2007 12:19:38 +0200 Subject: [postmodern-devel] Help with query replacement In-Reply-To: <46095593.2060202@gmail.com> References: <4608E616.1040507@gmail.com> <46095593.2060202@gmail.com> Message-ID: Hi, > By paging, I meant paging in the web pages - if I have 100 > results to a query, I would like to show that to the user in chunks of > 20 rows over 5 pages Ah, I see what you mean now. And I suppose it would be possible to write some kind of macro for that. Or make the query a real run-time list, and use compile-query to execute it. I don't have a lisp or very much time right now, so I won't help you write the functions, but you can bug the people in #lisp on irc about it or something if you can't figure it out. > Also, the element :count was not added to postmodern - I have checked > the source too. So, I have to use :raw for now - please correct me if > I'm wrong. Did you try (sql (:count '*))? It works. By magic. Actually, S-SQL has a fallback rule for operators it does not know. They are rendered as simple function calls -- function(arg1, arg2, ...) -- so things like :count and any stored procedures you might define work without the need for defining the operators. Regards, Marijn From marijnh at gmail.com Sat Mar 31 10:54:38 2007 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sat, 31 Mar 2007 12:54:38 +0200 Subject: [postmodern-devel] Bug in :limit, patch for s-sql.lisp In-Reply-To: <46062C0E.1040304@gmail.com> References: <46062C0E.1040304@gmail.com> Message-ID: Hi, I adjusted the patch a bit, it does not add an :offset operator, but does make the second argument to :limit work correctly. If you ever find yourself needing just :offset without :limit, let me know, and we can add that operator as well. I never came across such a situation though. The fixed :limit has been pushed into the darcs repo and will be included in the next release. Cheers, Marijn