From john.nmi.morrison at gmail.com Tue Jun 14 11:37:15 2011 From: john.nmi.morrison at gmail.com (John Morrison) Date: Tue, 14 Jun 2011 07:37:15 -0400 Subject: [postmodern-devel] using dao-class on database with column names like "addr:city" Message-ID: <201106140737.15932.john.nmi.morrison@gmail.com> Hi All; Sorry if this is a dumb newbie question. I have a PostGIS database with column names that are rather unfortunate (and created not by me but by/via many different bits of GIS software, and they sadly seem not to have mechanisms to specify rewriting of column names on the fly during the workflow). Some of the tables have column names which have colons in them. When I try and do the first Obvious Thing I can think of (name my dao-defclass slots e.g., |addr:city| in an attempt to match the column name), get-dao complains thusly: --- cut here --- No slot named addr:city in class CM-LOCATION. DAO out of sync with table, or incorrect query used. --- cut here A trip into the debugger leads me to believe that the slot name |addr:city| in the dao-class defclass gets rewritten as "addr_city", and then the get-dao from the table with column name "addr:city" (which does not seem to likewise get passed through to-sql-name, as it IS in fact an sql name) fails. (FWIW I have never had any trouble whatsoever with normally-named columns in almost all other databases I have processed.) Another Obvious Thing to try - setting *escape-sql-names-p* around the defclass (as suggested in another mailing to this list circa 2008) didn't seem to affect things either (but that could be Pilot Error). The final Obvious Thing to try was figuring out how to set the sql-name in the slot definition, but there seemed to be no initarg or other mechanisms I could spot. If some kind soul out there would point me in the right direction, I'd greatly appreciate it. Thanks in advance, -jm p.s., and then there are the database column names (e.g., "type") that are locked in various CL packages... -- --- John Morrison --- john.nmi.morrison at gmail.com From marijnh at gmail.com Tue Jun 14 11:59:03 2011 From: marijnh at gmail.com (Marijn Haverbeke) Date: Tue, 14 Jun 2011 13:59:03 +0200 Subject: [postmodern-devel] using dao-class on database with column names like "addr:city" In-Reply-To: <201106140737.15932.john.nmi.morrison@gmail.com> References: <201106140737.15932.john.nmi.morrison@gmail.com> Message-ID: > A trip into the debugger leads me to believe that the slot name > |addr:city| in the dao-class defclass gets rewritten as "addr_city", If colons are safe in SQL field names, you can change s-sql::to-sql-name to leave them intact (there seems to already by an exception for * characters). > p.s., and then there are the database column names (e.g., "type") that > are locked in various CL packages... Slot names can be locked? I've never heard of that. How? From john.nmi.morrison at gmail.com Tue Jun 14 12:33:34 2011 From: john.nmi.morrison at gmail.com (John Morrison) Date: Tue, 14 Jun 2011 08:33:34 -0400 Subject: [postmodern-devel] using dao-class on database with column names like "addr:city" In-Reply-To: References: <201106140737.15932.john.nmi.morrison@gmail.com> Message-ID: <201106140833.34306.john.nmi.morrison@gmail.com> On Tuesday 14 June 2011, Marijn Haverbeke wrote: > If colons are safe in SQL field names, you can change > s-sql::to-sql-name to leave them intact (there seems to already by an > exception for * characters). Thanks, and will do... As regards colons in column names: while I am by no means an experienced SQL guy, AFAICT the various GIS tools I am using all seem (so far) to be functioning correctly on databases with those column names. > Slot names can be locked? I've never heard of that. How? IIRC I got a package lock violation when naming the dao-class' slot "type" to mirror the SQL column name. I will go back and look again - only spent 2 seconds kludging around it, as I ran into the colon problem on the same table.... Thanks again for the help! -jm > > _______________________________________________ > postmodern-devel mailing list > postmodern-devel at common-lisp.net > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel -- --- John Morrison --- john.nmi.morrison at gmail.com From xach at xach.com Tue Jun 14 12:48:02 2011 From: xach at xach.com (Zach Beane) Date: Tue, 14 Jun 2011 08:48:02 -0400 Subject: [postmodern-devel] using dao-class on database with column names like "addr:city" In-Reply-To: <201106140833.34306.john.nmi.morrison@gmail.com> (John Morrison's message of "Tue, 14 Jun 2011 08:33:34 -0400") References: <201106140737.15932.john.nmi.morrison@gmail.com> <201106140833.34306.john.nmi.morrison@gmail.com> Message-ID: <8739jc60l9.fsf@hangup.portland.xach.com> John Morrison writes: > On Tuesday 14 June 2011, Marijn Haverbeke wrote: >> If colons are safe in SQL field names, you can change >> s-sql::to-sql-name to leave them intact (there seems to already by an >> exception for * characters). > > Thanks, and will do... > > As regards colons in column names: while I am by no means an > experienced SQL guy, AFAICT the various GIS tools I am using all seem > (so far) to be functioning correctly on databases with those column > names. For what it's worth, a column (or other object) may use any kind of string as its name if enclosed in double-quotes. They're a bit like CL symbols and ||s in that regard. db=> create table foo ("I am a Column" int); CREATE TABLE db=> insert into foo values (1); INSERT 0 1 db=> select * from foo; I am a Column --------------- 1 (1 row) Zach From rm at tuxteam.de Tue Jun 14 13:45:29 2011 From: rm at tuxteam.de (rm at tuxteam.de) Date: Tue, 14 Jun 2011 15:45:29 +0200 Subject: [postmodern-devel] using dao-class on database with column names like "addr:city" In-Reply-To: <201106140833.34306.john.nmi.morrison@gmail.com> References: <201106140737.15932.john.nmi.morrison@gmail.com> <201106140833.34306.john.nmi.morrison@gmail.com> Message-ID: <20110614134529.GA30010@seid-online.de> On Tue, Jun 14, 2011 at 08:33:34AM -0400, John Morrison wrote: > On Tuesday 14 June 2011, Marijn Haverbeke wrote: > > If colons are safe in SQL field names, you can change > > s-sql::to-sql-name to leave them intact (there seems to already by an > > exception for * characters). > > Thanks, and will do... > > As regards colons in column names: while I am by no means an > experienced SQL guy, AFAICT the various GIS tools I am using all seem > (so far) to be functioning correctly on databases with those column > names. Sorry, but colons aren't a avalid part of an SQL identifier. A is a character string, up to 128 characters long, that consists only of letters, digits, and underscore characters. It must begin with a letter. In standard-conforming SQL 'addr:city' only works as a delimited identifier: A is a character string, up to 128 characters long, surrounded by a pair of double quote marks. (The delimiting double quotes aren't part of the , so they're not included in the calculation of its size.) Two consecutive double quotes within the character string (i.e., "") represent one double quote mark; together, they count as one character when calculating the size of the . A is case sensitive. That is, uppercase and lowercase letters within a are not interchangeable; This smells like open streetmap ... must have been SQL gurus that designed that schema :-/ > > Slot names can be locked? I've never heard of that. How? > > IIRC I got a package lock violation when naming the dao-class' slot > "type" to mirror the SQL column name. I will go back and look again - > only spent 2 seconds kludging around it, as I ran into the colon > problem on the same table.... This seems wrong. The symbol 'type' only causes lock-problems when you try to bind its function value. Are you shure you didn't: (defclass foo () ((type :accessor type))) ??? That _would_ try to create a function '(type instance)' and trigger the package lock violation error. HTH Ralf Mattes > Thanks again for the help! > > -jm > > > > > _______________________________________________ > > postmodern-devel mailing list > > postmodern-devel at common-lisp.net > > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel > > > -- > --- John Morrison > --- john.nmi.morrison at gmail.com > > _______________________________________________ > postmodern-devel mailing list > postmodern-devel at common-lisp.net > http://lists.common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel From john.nmi.morrison at gmail.com Tue Jun 14 14:40:26 2011 From: john.nmi.morrison at gmail.com (John Morrison) Date: Tue, 14 Jun 2011 10:40:26 -0400 Subject: [postmodern-devel] =?iso-8859-1?q?using_dao-class_on_database_wit?= =?iso-8859-1?q?h_column=09names_like_=22addr=3Acity=22?= In-Reply-To: <20110614134529.GA30010@seid-online.de> References: <201106140737.15932.john.nmi.morrison@gmail.com> <201106140833.34306.john.nmi.morrison@gmail.com> <20110614134529.GA30010@seid-online.de> Message-ID: <201106141040.26825.john.nmi.morrison@gmail.com> On Tuesday 14 June 2011, rm at tuxteam.de wrote: > Sorry, but colons aren't a avalid part of an SQL identifier. I would be happy with any mechanism at all which would enable me to cope with such non-conforming databases. What's the Right Thing to do? > This smells like open streetmap ... must have been SQL gurus that > designed that schema :-/ Cloudmade.com produced the shapefiles (which presumably contain OSM data - not sure of the toolchain used). I used PostGIS 2.0 and its shp2pgsql (built from source) to put the shapefiles into PostGreSQL 9.0.4-2. Regarding OpenStreetMap: FYI when I was using Osmosis to import OSM data myself (into PostGIS 1.5x), I didn't see this problem as the schema was completely different and a lot simpler. And I cannot easily go back to Osmosis under PostGIS 2.0 as a lot of the PostGIS function names (upon which Osmosis relies) have changed. > This seems wrong. The symbol 'type' only causes lock-problems when you > try to bind its function value. Are you shure you didn't: > > (defclass foo () > ((type :accessor type))) At one point I am sure I did. Thanks for the help!! -jm -- --- John Morrison --- john.nmi.morrison at gmail.com From marijnh at gmail.com Tue Jun 14 14:59:24 2011 From: marijnh at gmail.com (Marijn Haverbeke) Date: Tue, 14 Jun 2011 16:59:24 +0200 Subject: [postmodern-devel] using dao-class on database with column names like "addr:city" In-Reply-To: <201106141040.26825.john.nmi.morrison@gmail.com> References: <201106140737.15932.john.nmi.morrison@gmail.com> <201106140833.34306.john.nmi.morrison@gmail.com> <20110614134529.GA30010@seid-online.de> <201106141040.26825.john.nmi.morrison@gmail.com> Message-ID: > I would be happy with any mechanism at all which would enable me to > cope with such non-conforming databases. ?What's the Right Thing to > do? The problem is that Postmodern went with a 'simplify Lisp symbols to get credible SQL names' scheme that assumes those credible SQL names are not escaped. Some major touching up would be necessary, both in the dao metaclass code and in s-sql, to make it possible to get around this assumption. From john.nmi.morrison at gmail.com Sat Jun 18 18:10:56 2011 From: john.nmi.morrison at gmail.com (John Morrison) Date: Sat, 18 Jun 2011 14:10:56 -0400 Subject: [postmodern-devel] VACUUM, ANALYZE Message-ID: <201106181410.56182.john.nmi.morrison@gmail.com> Hi All; I am using PostModern for PostGIS databases, and need to periodically VACUUM and/or ANALYZE the database. I ended up writing SQL ops for VACUUM and ANALYZE. There are some caveats which I mention in case their behavior should be changed and/or they should be rewritten to be more elegant. I am torn between embarrassment at the attached code and my desire to contribute to PostModern (which I find so useful). Obviously, I chose to brave embarrassment. In any event, advice, suggestions, and feedback are welcomed (in decreasing order). (1) The syntax I use does not follow the SQL spec. Instead of: VACUUM FREEZE VACUUM FULL FREEZE VERBOSE table0 column0 column1 I respectively do: (:vacuum :freeze) (:vacuum 'table0 'column0 'column1 :full :freeze :verbose) Please advise if this is the Wrong Thing to do, and if so, which arg-processing macros/helpers I should be using instead of the one I have perhaps, er, misused. (2) The split-on-keywords mechanism seems to enforce order of optional SQL arguments (i.e., FREEZE must precede VERBOSE). I am not sure that is either necessary or acceptable, or I am missing something. (3) The argument-sniffing at the end of the split-on-keywords is ugly. It sniffs the args to see if there are (non-keyword)table/column optional SQL arguments (implying a list of table, and optionally columns), and only sticks in a (:vars ...) if there are any. -jm -- --- John Morrison --- john.nmi.morrison at gmail.com -------------- next part -------------- (in-package :s-sql) (def-sql-op :vacuum (&rest args) (split-on-keywords ((vars * ?) (full - ?) (freeze - ?) (verbose - ?) (analyze - ?)) (cond ((null args) nil) ((and (listp args) (keywordp (first args))) args) (t (cons :vars args))) `("VACUUM " ,@(if full '("FULL ")) ,@(if freeze '("FREEZE ")) ,@(if verbose '("VERBOSE ")) ,@(if analyze '("ANALYZE ")) ,@(sql-expand-list vars)))) #| (sql (:vacuum)) (sql (:vacuum :full)) (sql (:vacuum :analyze)) (sql (:vacuum 't0 :full)) (sql (:vacuum 't0 'c0 'c1)) (sql (:vacuum 't0 :analyze)) (sql (:vacuum 't0 'c0 :analyze)) (sql (:vacuum :full :freeze :verbose :analyze)) (sql (:vacuum 't0 'c0 'c1 :full :freeze :verbose :analyze)) |# (def-sql-op :analyze (&rest args) (split-on-keywords ((vars * ?) (verbose - ?)) (cond ((null args) nil) ((and (listp args) (keywordp (first args))) args) (t (cons :vars args))) `("ANALYZE " ,@(if verbose '("VERBOSE ")) ,@(sql-expand-list vars)))) #| (sql (:analyze)) (sql (:analyze 't0)) (sql (:analyze 't0 'c0)) (sql (:analyze 't0 'c0 'c1)) (sql (:analyze :verbose)) (sql (:analyze 't0 :verbose)) (sql (:analyze 't0 'c0 :verbose)) (sql (:analyze 't0 'c0 'c1 :verbose)) |# From marijnh at gmail.com Wed Jun 22 11:26:25 2011 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 22 Jun 2011 13:26:25 +0200 Subject: [postmodern-devel] VACUUM, ANALYZE In-Reply-To: <201106181410.56182.john.nmi.morrison@gmail.com> References: <201106181410.56182.john.nmi.morrison@gmail.com> Message-ID: Hi Jim, Looks like these ops have a very simple structure. Maybe simply scanning the arguments for those keywords 'by hand' yields more pleasant code? That'll also make it easier to put the table and column names at the end. Best, Marijn From marijnh at gmail.com Wed Jun 22 11:27:01 2011 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 22 Jun 2011 13:27:01 +0200 Subject: [postmodern-devel] VACUUM, ANALYZE In-Reply-To: References: <201106181410.56182.john.nmi.morrison@gmail.com> Message-ID: Err, John. I half-read the -jm at the bottom and made Jim out of it. Sorry.