From lispnik at gmail.com Fri Jan 4 11:40:23 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Fri, 04 Jan 2008 17:40:23 +0600 Subject: [postmodern-devel] Inserting into related tables Message-ID: I don't quite understand how to do inserts into related table. For example, we have table 'items' with field 'producer_id' and table 'producer' with 'producer_id' and 'producer_name' fields: ,---- | (deftable item () | ((producer-id :type (or integer db-null)") | ... ; Other fields | ) | (:auto-id id)) | | (postmodern:deftable producers () | ((producer-name :type (string 50))) | (:auto-id producer-id) | (:indices producer)) | | ;; Add unique index manually | (postmodern:execute | (:create-unique-index 'producers-name-uniq-idx | :on 'producers | :fields 'producer-name)) `---- When we add new item to the database, producer may exist or may be new. We have to check producer's existense and add it to table if required. The only safe way to do it seems to be ,---- | ;; We ignore error if name exists. But what about other errors? | (ignore-errors | (let ((id (postmodern:next-id 'producers))) | (postmodern:execute | (:insert-into table :set | 'producer-id id | 'producer-name name)))) | | ;; Now get ID to insert into items.producer-id. | (postmodern:query | (:select 'producer-id :from table :where (:= 'producer-name name)) | :single) `---- However, on each insertion we increment producers-producer-id-seq even if producer name already exists in the table. This is somewhat clumsy. And faulty query wrapped with ignore-errors doesn't work with transactions. Is there any other postmodern-way? -- Ivan Boldyrev XML -- new language of ML family. From marijnh at gmail.com Fri Jan 4 14:35:07 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Fri, 4 Jan 2008 15:35:07 +0100 Subject: [postmodern-devel] Inserting into related tables In-Reply-To: References: Message-ID: Hi Ivan, Why don't you just start by looking for the record, and create a new record only if it is not found? Something like (untested): (defun get-producer-id (name) (or (query (:select 'id :from 'producer :where (:= 'name name))) On Jan 4, 2008 12:40 PM, Ivan Boldyrev wrote: > I don't quite understand how to do inserts into related table. > > For example, we have table 'items' with field 'producer_id' and table > 'producer' with 'producer_id' and 'producer_name' fields: > > ,---- > | (deftable item () > | ((producer-id :type (or integer db-null)") > | ... ; Other fields > | ) > | (:auto-id id)) > | > | (postmodern:deftable producers () > | ((producer-name :type (string 50))) > | (:auto-id producer-id) > | (:indices producer)) > | > | ;; Add unique index manually > | (postmodern:execute > | (:create-unique-index 'producers-name-uniq-idx > | :on 'producers > | :fields 'producer-name)) > `---- > > When we add new item to the database, producer may exist or may be new. > We have to check producer's existense and add it to table if required. > The only safe way to do it seems to be > > ,---- > | ;; We ignore error if name exists. But what about other errors? > | (ignore-errors > | (let ((id (postmodern:next-id 'producers))) > | (postmodern:execute > | (:insert-into table :set > | 'producer-id id > | 'producer-name name)))) > | > | ;; Now get ID to insert into items.producer-id. > | (postmodern:query > | (:select 'producer-id :from table :where (:= 'producer-name name)) > | :single) > `---- > > However, on each insertion we increment producers-producer-id-seq even > if producer name already exists in the table. This is somewhat clumsy. > And faulty query wrapped with ignore-errors doesn't work with transactions. > Is there any other postmodern-way? > > -- > Ivan Boldyrev > > XML -- new language of ML family. > _______________________________________________ > postmodern-devel mailing list > postmodern-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel > From marijnh at gmail.com Fri Jan 4 14:37:26 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Fri, 4 Jan 2008 15:37:26 +0100 Subject: [postmodern-devel] Inserting into related tables In-Reply-To: References: Message-ID: Ugh, previous message got sent before I finished typing. Here we go again... (defun get-producer-id (name) (or (query (:select 'id :from 'producer :where (:= 'name name))) (let ((new-producer (make-instance 'producer :name name)) (insert-dao new-producer) (get-id new-producer)))) Does that work (it is probably full of typos and mistakes, but you get the idea). Cheers, Marijn From marijnh at gmail.com Fri Jan 4 15:52:19 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Fri, 4 Jan 2008 16:52:19 +0100 Subject: [postmodern-devel] Inserting into related tables In-Reply-To: References: Message-ID: > (defun get-producer-id (name) > (or (query (:select 'id :from 'producer :where (:= 'name name))) > (let ((new-producer (make-instance 'producer :name name)) > (insert-dao new-producer) > (get-id new-producer)))) (Actually, now that I think about it, this would probably not be thread-safe. I'll see if I can figure out a better approach.) Cheers, Marijn From lispnik at gmail.com Fri Jan 4 16:01:46 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Fri, 04 Jan 2008 22:01:46 +0600 Subject: [postmodern-devel] Inserting into related tables In-Reply-To: (Marijn Haverbeke's message of "Fri\, 4 Jan 2008 15\:37\:26 +0100") References: Message-ID: On 10074 day of my life Marijn Haverbeke wrote: > (defun get-producer-id (name) > (or (query (:select 'id :from 'producer :where (:= 'name name))) > (let ((new-producer (make-instance 'producer :name name)) > (insert-dao new-producer) > (get-id new-producer)))) > > Does that work (it is probably full of typos and mistakes, but you get > the idea). There is a race condition: Session A Session B (:select ... "AA")) => nil (:select ... "AA")) => nil (sequence-next ...) => 10 (sequence-next ...) => 11 (:insert 'producer :set 'id 10 (:insert producer :set id 11 'name "AA") name "AA") So what is ID of "AA": 10 or 11? And if UNIQUE index is created, one of sessions will fail. If no UNIQUE index is created, we have two lines of "AA" in 'producers'. Perhaps, I misunderstand something, I'm quite new to SQL, transactons and so on. MySQL has REPLACE and INSERT IGNORE extensions. Perhaps, the only sane way of doing it in PostgreSQL is using locks, but they are another can of worms... -- Ivan Boldyrev Perl is a language where 2 x 2 is not equal to 4. From marijnh at gmail.com Fri Jan 4 16:15:01 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Fri, 4 Jan 2008 17:15:01 +0100 Subject: [postmodern-devel] Inserting into related tables In-Reply-To: References: Message-ID: > There is a race condition: You are entirely right, of course. I asked about this on #postgresql on IRC (freenode), and they didn't give me a better answer than your first try. Wasting IDs is rather ugly -- though I'm not sure this would be a concrete problem. Using ignore-errors is probably bad. Each postmodern exception has a database-error-code accessor, which can be used to identify a specific type of errors, and let all others through. Hope that helps. If you come up with a better way, let me know. Cheers, Marijn From marijnh at gmail.com Fri Jan 4 16:16:47 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Fri, 4 Jan 2008 17:16:47 +0100 Subject: [postmodern-devel] Inserting into related tables In-Reply-To: References: Message-ID: Oh (and I'm really sending too many separate mails today) -- have you tried just using the producer names as their IDs? This has worked quite well for me in the past (of course, if you have to add more information to producers you will still need the producer table anyway, but you do get rid of the useless ID numbers). Marijn From lispnik at gmail.com Fri Jan 4 17:09:23 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Fri, 04 Jan 2008 23:09:23 +0600 Subject: [postmodern-devel] Inserting into related tables In-Reply-To: (Marijn Haverbeke's message of "Fri\, 4 Jan 2008 17\:15\:01 +0100") References: Message-ID: <8eejcxy0u4.fsf@ibhome.cgitftp.uiggm.nsc.ru> On 10074 day of my life Marijn Haverbeke wrote: > Hope that helps. If you come up with a better way, let me know. I have googled for "postgres insert ignore" and found solution using SAVEPOINT and ROLLBACK TO: http://robbat2.livejournal.com/214267.html (The example is from PosgreSQL UPDATE documentation). On 10074 day of my life Marijn Haverbeke wrote: > Oh (and I'm really sending too many separate mails today) -- have you > tried just using the producer names as their IDs? It only solves wasted IDs problem, but no other, doesn't it? And joins on strings are slower than on ints (though I cannot say my application is so much performance-constrained). -- Ivan Boldyrev "Assembly of Japanese bicycle require great peace of mind." From marijnh at gmail.com Fri Jan 4 17:43:59 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Fri, 4 Jan 2008 18:43:59 +0100 Subject: [postmodern-devel] Inserting into related tables In-Reply-To: <8eejcxy0u4.fsf@ibhome.cgitftp.uiggm.nsc.ru> References: <8eejcxy0u4.fsf@ibhome.cgitftp.uiggm.nsc.ru> Message-ID: > I have googled for "postgres insert ignore" and found solution using > SAVEPOINT and ROLLBACK TO: > http://robbat2.livejournal.com/214267.html Does that actually solve the wasted id problem? > > have you tried just using the producer names as their IDs? > > It only solves wasted IDs problem, but no other, doesn't it? The idea was that you wouldn't need the table that maps names to ids at all, so this whole problem goes away. I'm not sure that joining on strings is really that much slowing in Postgres, but I haven't got any benchmarks to really find out. Anyway, I also came up with this monstrosity, it doesn't solve the wasted id problem, but take a look: (let ((name "dummy") (id (generate-next-id))) (execute (:insert-into 'producer (:except (:select name id) (:select '* :from 'producer :where (:= 'name name)))))) It uses EXCEPT to remove the new row from the set of inserted rows if it already exists. If you want to use it, you'll have to pull in a patch I just added to Postmodern because my :insert-into operator was a bit too picky about the input it allowed. Cheers, Marijn From lispnik at gmail.com Sat Jan 5 06:53:06 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Sat, 05 Jan 2008 12:53:06 +0600 Subject: [postmodern-devel] WITH-SAVEPOINT macro Message-ID: WITH-SAVEPOINT macro is modelled after WITH-TRANSACTION. I would be glad to see it in 1.05 :) -- Ivan Boldyrev Assembly of a Japanese bicycle requires greatest peace of spirit. -------------- next part -------------- A non-text attachment was scrubbed... Name: savepoint.lisp Type: text/x-common-lisp Size: 1636 bytes Desc: not available URL: From lispnik at gmail.com Fri Jan 4 18:49:26 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Sat, 05 Jan 2008 00:49:26 +0600 Subject: [postmodern-devel] Bug report: Incorrect handling of SIMPLE-DATE types in S-SQL Message-ID: <18fxxdfmtl.fsf@ibhome.cgitftp.uiggm.nsc.ru> This doesn't work: ,---- | (postmodern:query | (:select (:- :localtimestamp | (simple-date:encode-interval :hour 2 :minute 10)))) | | Database error 22007: invalid input syntax for type timestamp: "2 hours 10 minutes " | Query: (SELECT (localtimestamp - '2 hours 10 minutes ')) `---- This does work: ,---- | (postmodern:query | (:select (:- :localtimestamp (:raw "interval '2 hours 10 minutes'")))) `---- It was tested with PostgreSQL 8.0.13 and 8.2.4. SQL-IZE methods for SIMPLE-DATE objects should return escaped strings like "interval '2 hours 10 minutes'", not just "2 hours 10 minutes", and second value of SQL-IZE must be NIL: b ,---- | (:method ((arg simple-date:date)) | (multiple-value-bind (year month day) (simple-date:decode-date arg) | (format nil "date ~A" | (sql-escape-string (format nil "~4,'0d-~2,'0d-~2,'0d" year month day))))) `---- (Untested!) Perhaps, sql-escape-string can be omitted: the output has no dangerous characters anyway. ,---- | (:method ((arg simple-date:date)) | (multiple-value-bind (year month day) (simple-date:decode-date arg) | (format nil "date '~4,'0d-~2,'0d-~2,'0d'" year month day))) `---- -- Ivan Boldyrev Violets are red, Roses are blue. // I'm schizophrenic, And so am I. From marijnh at gmail.com Sat Jan 5 09:40:30 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sat, 5 Jan 2008 10:40:30 +0100 Subject: [postmodern-devel] WITH-SAVEPOINT macro In-Reply-To: References: Message-ID: > WITH-SAVEPOINT macro is modelled after WITH-TRANSACTION. I would be > glad to see it in 1.05 :) Looks good. Is it intentional that the savepoint is rolled back after its body finishes, unless the body explicitly released/committed the savepoint? Also, inheriting from transaction-handle means that abort-transaction and commit-transaction are also valid on savepoints. Does this make sense, or should we just create a whole new class for savepoints? Cheers, Marijn From marijnh at gmail.com Sat Jan 5 09:44:28 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sat, 5 Jan 2008 10:44:28 +0100 Subject: [postmodern-devel] Bug report: Incorrect handling of SIMPLE-DATE types in S-SQL In-Reply-To: <18fxxdfmtl.fsf@ibhome.cgitftp.uiggm.nsc.ru> References: <18fxxdfmtl.fsf@ibhome.cgitftp.uiggm.nsc.ru> Message-ID: So far I always used these types like (:select (:type my-date-value date)), but explicitly putting 'date' in front of the value string seems to work (I had never seen it before). Only, there is a reason that sql-ize allows one to take the un-escaped version of a value -- when passing arguments to queries with $1-like 'holes' in them, you need to pass them as un-escaped strings, and something like "date '2000-01-01'" produces an error. I've modified sql-escape-string to take an optional second parameter, a prefix, and when sql-ize returns a string as its second value (rather than T or NIL), this now means that the value should be escaped, and that string can be prefixed as type info. This seems to work fine (and has been applied to the repository). Cheers, Marijn From lispnik at gmail.com Sat Jan 5 10:01:19 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Sat, 05 Jan 2008 16:01:19 +0600 Subject: [postmodern-devel] Inserting into related tables In-Reply-To: (Marijn Haverbeke's message of "Fri\, 4 Jan 2008 18\:43\:59 +0100") References: <8eejcxy0u4.fsf@ibhome.cgitftp.uiggm.nsc.ru> Message-ID: On 10074 day of my life Marijn Haverbeke wrote: >> SAVEPOINT and ROLLBACK TO: >> http://robbat2.livejournal.com/214267.html > Does that actually solve the wasted id problem? No, "a nextval operation is never rolled back". >> > have you tried just using the producer names as their IDs? >> >> It only solves wasted IDs problem, but no other, doesn't it? > > The idea was that you wouldn't need the table that maps names to ids > at all, so this whole problem goes away. It was original design (created by some student). However, I need to do queries like "get all producers like 'abc%'" and "get all producers" for autocompletion and input validation. SELECT DISTINCT works, but look at the timings: CL-USER> (time (dotimes (i 100) (postmodern:query (:select (:distinct 'producer) :from 'catalog)))) Evaluation took: 3.353 seconds of real time 0.056004 seconds of user run time 0.016001 seconds of system run time 0 calls to %EVAL 0 page faults and 417,472 bytes consed. NIL CL-USER> (time (dotimes (i 100) (postmodern:query (:select 'producer :from 'producers)))) Evaluation took: 0.142 seconds of real time 0.032002 seconds of user run time 0.008001 seconds of system run time 0 calls to %EVAL 0 page faults and 273,816 bytes consed. NIL (Indicies are present). Well, for my application such difference is not really important, but what if it will in next project? > Anyway, I also came up with this monstrosity, it doesn't solve the > wasted id problem, but take a look: Thanks, I will note. But so far SAVEPOINT solution looks best for me. -- Ivan Boldyrev Violets are red Roses are blue It's amazing what DNA splicing can do. From lispnik at gmail.com Sat Jan 5 10:09:01 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Sat, 05 Jan 2008 16:09:01 +0600 Subject: [postmodern-devel] Bug report: Incorrect handling of SIMPLE-DATE types in S-SQL In-Reply-To: (Marijn Haverbeke's message of "Sat\, 5 Jan 2008 10\:44\:28 +0100") References: <18fxxdfmtl.fsf@ibhome.cgitftp.uiggm.nsc.ru> Message-ID: On 10075 day of my life Marijn Haverbeke wrote: > So far I always used these types like (:select (:type my-date-value > date))... Ouch, I didn't new of such way. -- Ivan Boldyrev Your bytes are bitten. From lispnik at gmail.com Sat Jan 5 10:07:12 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Sat, 05 Jan 2008 16:07:12 +0600 Subject: [postmodern-devel] WITH-SAVEPOINT macro In-Reply-To: (Marijn Haverbeke's message of "Sat\, 5 Jan 2008 10\:40\:30 +0100") References: Message-ID: On 10075 day of my life Marijn Haverbeke wrote: > Looks good. Is it intentional that the savepoint is rolled back after > its body finishes, unless the body explicitly released/committed the > savepoint? Sorry, this is a last-minute-introduced bug. Swap release-savepoint and rollback-savepoint in with-savepoint definition. > Also, inheriting from transaction-handle means that abort-transaction > and commit-transaction are also valid on savepoints. Does this make > sense, ...? No. I subclassed transaction-handle just for code writing speed. If you think it is important, make savepoint-handle an independent class. -- Ivan Boldyrev Assembly of a Japanese bicycle requires greatest peace of spirit. From marijnh at gmail.com Sat Jan 5 11:04:05 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sat, 5 Jan 2008 12:04:05 +0100 Subject: [postmodern-devel] WITH-SAVEPOINT macro In-Reply-To: References: Message-ID: Okay, I think I fixed the code, and I added documentation. Could you see whether the current version works for you? If so, I will release it as a new version. Cheers, Marijn From lispnik at gmail.com Sat Jan 5 13:03:44 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Sat, 05 Jan 2008 19:03:44 +0600 Subject: [postmodern-devel] String escaping (was: WITH-SAVEPOINT macro) In-Reply-To: (Marijn Haverbeke's message of "Sat\, 5 Jan 2008 12\:04\:05 +0100") References: Message-ID: On 10075 day of my life Marijn Haverbeke wrote: > Okay, I think I fixed the code, and I added documentation. Could you > see whether the current version works for you? If so, I will release > it as a new version. My code seems to work fine with new version. But wait a minute. :) CL-USER> (postmodern:query (:select "\\'")) WARNING: Postgres warning: nonstandard use of \\ in a string literal Use the escape string syntax for backslashes, e.g., E'\\'. (("\\'")) SQL-IZE escapes #\' and #\\, however PosgreSQL (unlike MySQL) doesn't like \ in ordinary string, but have special E'...' syntax. Of course, warnings can be just ignored, and I can also supress this warning with setting standard_conforming_strings off, but isn't it better to output all string literals with E'...' syntax? Just add one letter :) -- Ivan Boldyrev Perl is a language where 2 x 2 is not equal to 4. From lispnik at gmail.com Sun Jan 6 17:03:00 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Sun, 06 Jan 2008 23:03:00 +0600 Subject: [postmodern-devel] Binary and unary operators Message-ID: 1. You are missing such operators like || and && (former is string concatenation, latter is tests overlapping geometric objects). There are also a. math: !, !!, #, >>, <<. b. geometric: ##, @@, @-@, ~=, <->, &<, &>, <<|, |>>, &<|, |&>, <^, >^, ?#, ?|, ?-|, ?||, @>, <@. c. network: <<=, >>=. ?| is both binary and unary operator, ! is postfix (but is equivalent to prefix !!, so you may define :! that expands to prefix !! or just :!! that expands to !!). There are also another operators like @, but they have functions-substitutions (@ x <=> abs(x)). psql's command \do (describe operators) also lists other operators. 2. As PSQL allows defining own operators, I think it worth exporting s-sql::def-infix-ops or creating some more flexible user macro (one might prefer to use :concat instead of :\|\|). With such facility you don't need to define such rare-used operators like @-@ and ?-|, allowing user to define them. -- Ivan Boldyrev Violets are red Roses are blue It's amazing what DNA splicing can do. From marijnh at gmail.com Sun Jan 6 20:55:48 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sun, 6 Jan 2008 21:55:48 +0100 Subject: [postmodern-devel] String escaping (was: WITH-SAVEPOINT macro) In-Reply-To: References: Message-ID: > My code seems to work fine with new version. Great, I will consider it functional then. > SQL-IZE escapes #\' and #\\, however PosgreSQL (unlike MySQL) doesn't > like \ in ordinary string, but have special E'...' syntax. Of course, > warnings can be just ignored, and I can also supress this warning with > setting standard_conforming_strings off, but isn't it better to output > all string literals with E'...' syntax? Just add one letter :) Done. I also added a *standard-sql-strings* variable which can be used to configure S-SQL to just use standard strings. It would have been nicer if this could just be derived directly from the connection, but at the moment S-SQL does not have a concept of connections, and that is kind of convenient, so if you really want to use standard strings you'll just have to change this parameter. Cheers, Marijn From marijnh at gmail.com Sun Jan 6 20:57:30 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sun, 6 Jan 2008 21:57:30 +0100 Subject: [postmodern-devel] Binary and unary operators In-Reply-To: References: Message-ID: Hey Ivan, > 1. You are missing such operators like || and && (former is string > concatenation, latter is tests overlapping geometric objects). You've probably noticed how easy it is to add operators to S-SQL. Patches defining additional operators that are commonly used are welcome -- || would be a good candidate. > 2. As PSQL allows defining own operators, I think it worth exporting > s-sql::def-infix-ops or creating some more flexible user macro (one > might prefer to use :concat instead of :\|\|). I very much agree. Wanna write a patch? ;) Cheers, Marijn From lispnik at gmail.com Mon Jan 7 14:50:38 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Mon, 07 Jan 2008 20:50:38 +0600 Subject: [postmodern-devel] Binary and unary operators In-Reply-To: (Marijn Haverbeke's message of "Sun\, 6 Jan 2008 21\:57\:30 +0100") References: Message-ID: On 10077 day of my life Marijn Haverbeke wrote: > ... a patch? Comment string may require full rewriting. diff -rN -u old-postmodern/postmodern/package.lisp new-postmodern/postmodern/package.lisp --- old-postmodern/postmodern/package.lisp 2008-01-07 20:38:27.000000000 +0600 +++ new-postmodern/postmodern/package.lisp 2008-01-07 20:38:27.000000000 +0600 @@ -25,6 +25,7 @@ #:smallint #:bigint #:numeric #:real #:double-precision #:bytea #:text #:varchar #:*escape-sql-names-p* #:sql-escape-string + #:def-infix-ops ;; Condition type from cl-postgres #:database-error #:database-error-message #:database-error-code diff -rN -u old-postmodern/s-sql/s-sql.lisp new-postmodern/s-sql/s-sql.lisp --- old-postmodern/s-sql/s-sql.lisp 2008-01-07 20:38:27.000000000 +0600 +++ new-postmodern/s-sql/s-sql.lisp 2008-01-07 20:38:27.000000000 +0600 @@ -16,6 +16,7 @@ #:to-sql-name #:sql-ize #:*escape-sql-names-p* + #:def-infix-ops #:sql #:sql-compile #:enable-s-sql-syntax)) @@ -373,21 +374,58 @@ (destructuring-bind ,arglist ,args-name , at body)))) -(defun expand-infix-op (operator allow-unary args) - (if (cdr args) - `("(" ,@(sql-expand-list args (strcat " " operator " ")) ")") - (if allow-unary - (sql-expand (first args)) - (error "SQL operator ~A takes at least two arguments." operator)))) - -(defmacro def-infix-ops (allow-unary &rest ops) +(defun expand-infix-op (operator class args) + (declare (type (member t :both nil) class)) + (cond + ((cdr args) + `("(" ,@(sql-expand-list args (strcat " " operator " ")) ")")) + ((eq class t) + (sql-expand (first args))) + ((eq class :both) + `(,operator "(" ,@(sql-expand (first args)) ")")) + (t + (error "SQL operator ~A takes at least two arguments." operator)))) + +(defmacro def-infix-ops (class &rest ops) + (declare (type (member t :both nil) class)) + "Define infix operators. +CLASS may be either T, :BOTH or NIL. + + 1. T. S-SQL operators may be both binary and unary, but unary form + is equivalent to the only argument itself (e.g. + (:AND condition) => \"condition\"). + + 2. :BOTH. S-SQL operators may be both binary and unary, but unlike + T, operator is kept before the argument in unary form. Example + is PosgreSQL's ?| operator: + (def-infix-ops :both :?\\|) + (:?\\| a) => \"?|(a)\" + (:?\\| a b) => \"a ?| b\". + + 3. NIL. Operator is binary only. + +OPS is a list of operator designators. There are two kinds of +operator designators: + + 1. A keyword. Downcased symbol value of the keword is used as SQL + name of operator. + + 2. Two-element list: (KEYWORD STRING). String is used as SQL name + of operator, and KEYWORD is S-SQL name of operator. + Example: (:concat \"||\")." `(progn ,@(mapcar (lambda (op) - `(defmethod expand-sql-op ((op (eql ,op)) args) - (expand-infix-op ,(string-downcase (symbol-name op)) ,allow-unary args))) + (when (keywordp op) + (setf op (list op (string-downcase (symbol-name op))))) + (let ((kwd (first op)) + (txt (second op))) + `(defmethod expand-sql-op ((op (eql ,kwd)) args) + (expand-infix-op ,txt ,class args)))) ops))) -(def-infix-ops t :+ :* :& :|\|| :and :or :union) -(def-infix-ops nil := :/ :!= :< :> :<= :>= :^ :intersect :except :~* :!~ :!~* :like :ilike) + +(def-infix-ops t :+ :* :& :|\|| :and :or :union :\|\| (:concat "||")) +(def-infix-ops nil := :/ :!= :< :> :<= :>= :^ :intersect :except :~* :!~ :!~* + :like :ilike :&& :% :\# :<< :>>) (def-sql-op :- (first &rest rest) (if rest -- Ivan Boldyrev Life! Don't talk to me about life. From lispnik at gmail.com Mon Jan 7 15:44:11 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Mon, 07 Jan 2008 21:44:11 +0600 Subject: [postmodern-devel] Binary and unary operators In-Reply-To: (Ivan Boldyrev's message of "Mon\, 07 Jan 2008 20\:50\:38 +0600") References: Message-ID: On 10077 day of my life Ivan Boldyrev wrote: > On 10077 day of my life Marijn Haverbeke wrote: >> ... a patch? > +(defmacro def-infix-ops (class &rest ops) BTW: names of definition macros with minus in their names are traditionally started with define, not def (define-modify-macro, define-method-combination, but defvar, defun, defmacro). So proper name for public macro should be define-infix-ops. -- Ivan Boldyrev Violets are red, Roses are blue. // I'm schizophrenic, And so am I. From lispnik at gmail.com Mon Jan 7 17:36:01 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Mon, 07 Jan 2008 23:36:01 +0600 Subject: [postmodern-devel] Performance patch Message-ID: <7xejctmtby.fsf@ibhome.cgitftp.uiggm.nsc.ru> My application uses quite long and complex query, and profiling showed that slowest function is SQL-COMPILE. First I reimplemented it with S-SQL::STRCAT, but then I thought: what if it will hit CALL-ARGUMENTS-LIMIT (it is only 4096 on GNU Clisp!)? So I changed implementation of S-SQL:STRCAT to get list as the only argument, not as &rest. SQL-COMPILE is much faster and consumes on my query 10 times less memory. As SQL-COMPILE is usually called on run time, not compilation time, I think it is very important. --- old/s-sql/s-sql.lisp~ 2008-01-07 20:05:17.000000000 +0600 +++ new/s-sql/s-sql.lisp 2008-01-07 23:26:03.000000000 +0600 @@ -25,7 +25,7 @@ ;; Utils -(defun strcat (&rest args) +(defun strcat (args) "Concatenate a list of strings into a single one." (let ((result (make-string (reduce #'+ args :initial-value 0 :key 'length)))) (loop :for pos = 0 :then (+ pos (length arg)) @@ -36,11 +36,11 @@ (defun implode (sep list) "Reduce a list of strings to a single string, inserting a separator between them." - (apply 'strcat - (loop :for element :on list - :collect (car element) - :if (cdr element) - :collect sep))) + (strcat + (loop :for element :on list + :collect (car element) + :if (cdr element) + :collect sep))) (defun split-on-keywords% (shape list) "Helper function for split-on-keywords. Extracts the values @@ -338,11 +338,11 @@ (let ((list (reduce-strings (sql-expand form)))) (if (= 1 (length list)) (car list) - `(strcat , at list)))) + `(strcat (list , at list))))) (defun sql-compile (form) (let ((*expand-runtime* t)) - (car (reduce-strings (sql-expand form))))) + (strcat (sql-expand form)))) ;; The reader syntax. @@ -378,7 +378,7 @@ (declare (type (member t :both nil) class)) (cond ((cdr args) - `("(" ,@(sql-expand-list args (strcat " " operator " ")) ")")) + `("(" ,@(sql-expand-list args (strcat (list " " operator " "))) ")")) ((eq class t) (sql-expand (first args))) ((eq class :both) -- Ivan Boldyrev Life! Don't talk to me about life. From marijnh at gmail.com Mon Jan 7 17:53:56 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Mon, 7 Jan 2008 18:53:56 +0100 Subject: [postmodern-devel] Performance patch In-Reply-To: <7xejctmtby.fsf@ibhome.cgitftp.uiggm.nsc.ru> References: <7xejctmtby.fsf@ibhome.cgitftp.uiggm.nsc.ru> Message-ID: > SQL-COMPILE is much faster and consumes on my query 10 times less > memory. As SQL-COMPILE is usually called on run time, not compilation > time, I think it is very important. Great find, I will try to apply it... but could you *please* use darcs to create patch files? I always have strange problems with raw diff files, and as an added advantage patches you record with darcs will automatically have your name associated with them. Just (assuming you have a darcs checkout) record your patches (with 'darcs record') and then type 'darcs send -o somefile.patch' and it will ask you which changes to include in the patch file. Then mail me the file, and applying your patches will be a breeze. Cheers, Marijn From marijnh at gmail.com Tue Jan 8 20:02:22 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Tue, 8 Jan 2008 21:02:22 +0100 Subject: [postmodern-devel] Binary and unary operators In-Reply-To: References: Message-ID: Ivan, I pushed patches making a register-sql-operators macro available and changing the way strings are passed to strcat. Take a look. Cheers, Marijn From lispnik at gmail.com Wed Jan 9 16:57:25 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Wed, 09 Jan 2008 22:57:25 +0600 Subject: [postmodern-devel] Binary and unary operators In-Reply-To: (Marijn Haverbeke's message of "Tue\, 8 Jan 2008 21\:02\:22 +0100") References: Message-ID: On 10079 day of my life Marijn Haverbeke wrote: > I pushed patches making a register-sql-operators macro available and > changing the way strings are passed to strcat. Take a look. Excellent, but you forgot to insert comma at symbol NAME in a :unary branch of MAKE-EXPANDER. ====================================================================== I tried to send appropriate one-line patch with darcs mail, but darcs doesn't allow sending only this particular patch. This is not a first time I stuck into such darcs' behaviour, and this is one of reasons I dislike this VCS. $ darcs send -u -i -o comma.patch ../postmodern Creating patch to "../postmodern"... Tue Jan 8 15:21:32 NOVT 2008 Ivan Boldyrev * Enchance infix operators definition macro. def-infix-ops is renamed to define-infix-ops, and its first argument now accepts :both. expand-infix-op is changed accordingly. Some new infix operators are added. Shall I send this patch? (1/4) [ynWvpxqadjk], or ? for help: n Tue Jan 8 15:22:01 NOVT 2008 Ivan Boldyrev * Make SQL generation faster. Shall I send this patch? (2/4) [ynWvpxqadjk], or ? for help: n You don't want to send any patches, and that's fine with me! ====================================================================== -- Ivan Boldyrev Today is the first day of the rest of your life. From marijnh at gmail.com Wed Jan 9 17:54:12 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 9 Jan 2008 18:54:12 +0100 Subject: [postmodern-devel] Binary and unary operators In-Reply-To: References: Message-ID: Hey Ivan, That's what I get for making 'trivial' changes just before committing something. I pushed a fix. The reason darcs doesn't prompt you to send that patch is (I think) that you have a bunch of other patches in your repository that are not in mine, and this change depends on one of those. Cheers, Marijn From marijnh at gmail.com Fri Jan 11 23:02:56 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sat, 12 Jan 2008 00:02:56 +0100 Subject: [postmodern-devel] Release 1.05 Message-ID: Hello list, Thanks to Ivan's relentless bug finding and feature requesting, version 1.05 of Postmodern is now a reality. See project page (http://common-lisp.net/project/postmodern/) or darcs log (http://common-lisp.net/cgi-bin/darcsweb/darcsweb.cgi?r=postmodern-postmodern;a=summary) for a change list. Enjoy, Marijn From airbaggins at gmail.com Mon Jan 14 08:19:30 2008 From: airbaggins at gmail.com (William Robinson) Date: Mon, 14 Jan 2008 08:19:30 +0000 Subject: [postmodern-devel] deftable class-name and unique indices problems Message-ID: <478B1B12.70400@gmail.com> Hi. Thanks for the library. I was struggling a bit with pomo's DAO support. A few points: I think it is worth reinforcing in the deftable documentation that a class-name NEEDS to be specified for the data-access methods to function. Can we have/is there a facility for creating unique multi-column indices in deftable? Some more examples for deftable would certainly not go amiss, particularly with repect to usage of :auto-id, :indices and :defer-id. Thanks Bill From marijnh at gmail.com Mon Jan 14 15:06:18 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Mon, 14 Jan 2008 16:06:18 +0100 Subject: [postmodern-devel] deftable class-name and unique indices problems In-Reply-To: <478B1B12.70400@gmail.com> References: <478B1B12.70400@gmail.com> Message-ID: Hey William, I retouched the deftable documentation a bit, hopefully making it clearer. There was no facility for directly cleating unique indices from deftable, but I have added a patch to the repository that makes this possible -- using "(:unique field-1 field-1)" instead of just "(field-1 field-2)" after the :indices keyword. If you have any specific suggestions for the docs, let me know, and I can incorporate them. Cheers, Marijn From lispnik at gmail.com Wed Jan 16 17:06:09 2008 From: lispnik at gmail.com (Ivan Boldyrev) Date: Wed, 16 Jan 2008 23:06:09 +0600 Subject: [postmodern-devel] Release 1.05 In-Reply-To: (Marijn Haverbeke's message of "Sat\, 12 Jan 2008 00\:02\:56 +0100") References: Message-ID: On 10082 day of my life Marijn Haverbeke wrote: > version 1.05 of Postmodern is now a reality. Very nice :) Btw, I would use different specifiers for char and varchar: (char n) => char(n) (string n) => varchar(n) To me, STRING is closer to variable-length entities than to fixed-length entities. Especially if you compare char[n] and string in C or Pascal :) -- Ivan Boldyrev Violets are red, Roses are blue. // I'm schizophrenic, And so am I. From marijnh at gmail.com Wed Jan 16 17:51:39 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 16 Jan 2008 18:51:39 +0100 Subject: [postmodern-devel] Release 1.05 In-Reply-To: References: Message-ID: On Jan 16, 2008 6:06 PM, Ivan Boldyrev wrote: > Btw, I would use different specifiers for char and varchar: > (char n) => char(n) > (string n) => varchar(n) That was also my first idea, but there are two objections: Firstly, Lisp values of type (string n) have to have *exactly* n characters, and are thus more like char(n) than varchar(n). Secondly, char is an exported symbol in the package :common-lisp, and thus not a very good candidate for use in a library (you aren't allowed to define a type by that name without first shadowing the symbol). Marijn From rm at seid-online.de Wed Jan 23 20:05:59 2008 From: rm at seid-online.de (Ralf Mattes) Date: Wed, 23 Jan 2008 21:05:59 +0100 Subject: [postmodern-devel] dynamically built queries Message-ID: <1201118759.6570.21.camel@localhost.localdomain> Hello list, I need to create queries dynamically and seem to have run into a silly showstopper. I want to create queries like ... SELECT tmp1.name, tmp1.value as "{foo}bar" from baz as tmp1; ... where both the table name alias ("tmp1") as well as the field name alias ("{foo}bar") are _Strings_. Now, if i do: (sql-compile '(:dot "tmp1" value)) --> "E'tmp1'.value" which is definitely _not_ what i want. The same for: (sql-compile '(:as "tmp1" baz)) --> "E'tmp1' AS baz" If possible at all I'd like to avoid the (make-symbol "tmp1") and similar workarrounds. Any ideas? TIA Ralf Mattes From marijnh at gmail.com Wed Jan 23 20:16:57 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 23 Jan 2008 21:16:57 +0100 Subject: [postmodern-devel] dynamically built queries In-Reply-To: <1201118759.6570.21.camel@localhost.localdomain> References: <1201118759.6570.21.camel@localhost.localdomain> Message-ID: Hi Ralf, I think that the :raw keyword can help you out here -- you give it a string, which is then inserted straight into the query, as in (sql (:select (:raw "tmp1.name") :from (:as 'baz (:raw "tmp1")))). Cheers, Marijn On Jan 23, 2008 9:05 PM, Ralf Mattes wrote: > Hello list, > > I need to create queries dynamically and seem to have run into a silly > showstopper. I want to create queries like ... > > SELECT tmp1.name, tmp1.value as "{foo}bar" from baz as tmp1; > > ... where both the table name alias ("tmp1") as well as the field name > alias ("{foo}bar") are _Strings_. Now, if i do: > > (sql-compile '(:dot "tmp1" value)) > --> "E'tmp1'.value" > > which is definitely _not_ what i want. The same for: > > (sql-compile '(:as "tmp1" baz)) > --> "E'tmp1' AS baz" > > If possible at all I'd like to avoid the (make-symbol "tmp1") and > similar workarrounds. Any ideas? > > TIA Ralf Mattes > > > _______________________________________________ > postmodern-devel mailing list > postmodern-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel >