From superdog at budzianowski.net Tue Sep 9 23:33:17 2008 From: superdog at budzianowski.net (Roman Budzianowski) Date: Tue, 9 Sep 2008 16:33:17 -0700 Subject: [postmodern-devel] character encoding Message-ID: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> I am trying to retrieve records from a db with SQL_ASCII. Any advice on how to deal with it? I don't see a way to change the encoding for a connection. Invalid byte at start of character: 0x99 [Condition of type TRIVIAL-UTF-8:UTF-8-DECODING-ERROR] I don't think I can change the encoding of the db. Roman From superdog at budzianowski.net Wed Sep 10 05:18:00 2008 From: superdog at budzianowski.net (Roman Budzianowski) Date: Tue, 9 Sep 2008 22:18:00 -0700 Subject: [postmodern-devel] character encoding In-Reply-To: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> References: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> Message-ID: <932373DB-0468-4F04-8961-AFD57CAD801E@budzianowski.net> Sorry, I misunderstood the design. CL-USER> (pomo:query "SHOW client_encoding") (("UNICODE")) It should work, so I why am I getting this error? On Sep 9, 2008, at 4:33 PM, Roman Budzianowski wrote: > I am trying to retrieve records from a db with SQL_ASCII. Any advice > on how to deal with it? > I don't see a way to change the encoding for a connection. > > Invalid byte at start of character: 0x99 > [Condition of type TRIVIAL-UTF-8:UTF-8-DECODING-ERROR] > > I don't think I can change the encoding of the db. > > Roman > > _______________________________________________ > postmodern-devel mailing list > postmodern-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel From marijnh at gmail.com Wed Sep 10 07:40:04 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 10 Sep 2008 09:40:04 +0200 Subject: [postmodern-devel] character encoding In-Reply-To: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> References: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> Message-ID: Hello Roman, > Invalid byte at start of character: 0x99 > [Condition of type TRIVIAL-UTF-8:UTF-8-DECODING-ERROR] This seems to suggest that the text in the record is not actually ASCII, but some other 8-bit encoding. 0x7f is the highest character code ASCII text can contain. What does 'psql -l' show? Actually, the way I understood it is that if you set a UNICODE/UTF-8 encoding for the connection, which Postmodern does, it should translate any text into that encoding before sending it... but that would mean the error you're seeing could never occur. I've never seen it happen before... switching your database to UNICODE encoding is bound to fix it, or you can try setting Postmodern to use 8-bit encoding (with the 0x7f-0xff characters being interpreted in whichever way your Lisp implementation feels like, most likely latin-1) by changing the *unicode* parameter in the cl-postgres.asd file, and re-compiling the library. Best, Marijn From superdog at budzianowski.net Wed Sep 10 15:46:38 2008 From: superdog at budzianowski.net (Roman Budzianowski) Date: Wed, 10 Sep 2008 08:46:38 -0700 Subject: [postmodern-devel] character encoding In-Reply-To: References: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> Message-ID: <7B64EEF3-FA8D-4103-8212-A918B9456B69@budzianowski.net> Marijn, I vaguely remember that I ran into it before, but since I was creating a new db I just switched to Unicode on the postgres side. I tried the same query in psql after setting the encoding to Unicode: NCS=# \encoding SQL_ASCII NCS=# \encoding UNICODE NCS=# \encoding UTF8 The query worked. Not sure how much of a proof it is, but it seems to imply that the problem might be on the lisp side. E.g. a byte might be skipped or misinterpreted. Roman On Sep 10, 2008, at 12:40 AM, Marijn Haverbeke wrote: > Hello Roman, > >> Invalid byte at start of character: 0x99 >> [Condition of type TRIVIAL-UTF-8:UTF-8-DECODING-ERROR] > > This seems to suggest that the text in the record is not actually > ASCII, but some other 8-bit encoding. 0x7f is the highest character > code ASCII text can contain. What does 'psql -l' show? Actually, the > way I understood it is that if you set a UNICODE/UTF-8 encoding for > the connection, which Postmodern does, it should translate any text > into that encoding before sending it... but that would mean the error > you're seeing could never occur. I've never seen it happen before... > switching your database to UNICODE encoding is bound to fix it, or you > can try setting Postmodern to use 8-bit encoding (with the 0x7f-0xff > characters being interpreted in whichever way your Lisp implementation > feels like, most likely latin-1) by changing the *unicode* parameter > in the cl-postgres.asd file, and re-compiling the library. > > Best, > Marijn > _______________________________________________ > postmodern-devel mailing list > postmodern-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel From marijnh at gmail.com Wed Sep 10 15:51:55 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 10 Sep 2008 17:51:55 +0200 Subject: [postmodern-devel] character encoding In-Reply-To: <7B64EEF3-FA8D-4103-8212-A918B9456B69@budzianowski.net> References: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> <7B64EEF3-FA8D-4103-8212-A918B9456B69@budzianowski.net> Message-ID: Hey, > it seems to imply that the problem might be on the lisp side. > E.g. a byte might be skipped or misinterpreted. Could be, but I can't imagine where. It would be useful if you could send me (some subset of) the DB that causes the error + the query you were trying, so I can try to reproduce the problem. Best, Marijn From superdog at budzianowski.net Wed Sep 10 16:40:04 2008 From: superdog at budzianowski.net (Roman Budzianowski) Date: Wed, 10 Sep 2008 09:40:04 -0700 Subject: [postmodern-devel] character encoding In-Reply-To: References: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> <7B64EEF3-FA8D-4103-8212-A918B9456B69@budzianowski.net> Message-ID: <30B7AC18-772B-4929-8356-9CE6E9F427DC@budzianowski.net> Marijn, I can retrieve some records. I am attaching a file with a record that failed in lisp but was retrieved fine in psql - no guarantee of course that the bytes are exactly the same. It failed in lisp like so: Invalid byte at start of character: 0xae [Condition of type TRIVIAL-UTF-8:UTF-8-DECODING-ERROR] Restarts: 0: [ABORT] Return to SLIME's top level. 1: [ABORT] Abort entirely from this (lisp) process. Backtrace: 0: (SWANK:SWANK-DEBUGGER-HOOK # #) 1: (ERROR TRIVIAL-UTF-8:UTF-8-DECODING-ERROR :BYTE 174 :MESSAGE "Invalid byte at start of character: 0x~X") Locals: EXCL::DATUM = TRIVIAL-UTF-8:UTF-8-DECODING-ERROR EXCL::ARGUMENTS = (:BYTE 174 :MESSAGE "Invalid byte at start of character: 0x~X") POSTMODERN::STATEMENT-ID = :STATEMENT-1 #:G1332 = WEB-SOURCE-ITEMS PQ-RDB::ARGS = (25) #:G1333 = WEB-SOURCE-ITEMS-SELECT-BY-ID PQ-RDB::SELECT-QUERY = "(SELECT * FROM web_source_items WHERE (id = $1))" ID = NIL POSTMODERN::PARAMS = (22225) EXCL::LOCAL-0 = (:BYTE 174 :MESSAGE "Invalid byte at start of character: 0x~X") EXCL::LOCAL-1 = TRIVIAL-UTF-8:UTF-8-DECODING-ERROR EXCL::LOCAL-2 = # EXCL::LOCAL-3 = 97 EXCL::LOCAL-4 = T EXCL::LOCAL-5 = 0 EXCL::LOCAL-6 = 8 :UNKNOWN = NIL :UNKNOWN = NIL :UNKNOWN = NIL 2: (TRIVIAL-UTF-8:UTF-8-GROUP-SIZE 174) Locals: BYTE = 174 POSTMODERN::STATEMENT-ID = :STATEMENT-1 #:G1332 = WEB-SOURCE-ITEMS PQ-RDB::ARGS = (25) #:G1333 = WEB-SOURCE-ITEMS-SELECT-BY-ID PQ-RDB::SELECT-QUERY = "(SELECT * FROM web_source_items WHERE (id = $1))" ID = NIL POSTMODERN::PARAMS = (22225) EXCL::LOCAL-0 = 174 3: (TRIVIAL-UTF-8:READ-UTF-8-STRING # :BYTE- LENGTH 1353) 4: ((:INTERNAL (:TOP-LEVEL-FORM "interpret.lisp" 4106) 0) # 1353) 5: ((FLET POSTMODERN::SYMBOL-ALIST-ROW-READER CL-POSTGRES:NEXT- FIELD) #) 6: (POSTMODERN::SYMBOL-ALIST-ROW-READER # #(# # # # # # ...)) 7: (CL-POSTGRES::SEND-EXECUTE # "STATEMENT-1" (22225) POSTMODERN::SYMBOL-ALIST-ROW-READER) 8: ((LABELS CL-POSTGRES:EXEC-PREPARED #:G3648)) 9: (CL-POSTGRES:EXEC-PREPARED # "STATEMENT-1" (22225) POSTMODERN::SYMBOL-ALIST-ROW-READER) 10: ((:INTERNAL WEB-SOURCE-ITEMS-SELECT-BY-ID) 22225) 11: (LET* ((PQ-RDB::SELECT-QUERY (GET # #)) (PQ-RDB::SELECT-FUNCTION (OR # #))) (APPLY PQ-RDB::SELECT-FUNCTION PQ-RDB::ARGS)) 12: (WEB-SOURCE-ITEMS-SELECT-BY-ID 22225) -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: 22225.txt URL: -------------- next part -------------- On Sep 10, 2008, at 8:51 AM, Marijn Haverbeke wrote: > Hey, > >> it seems to imply that the problem might be on the lisp side. >> E.g. a byte might be skipped or misinterpreted. > > Could be, but I can't imagine where. It would be useful if you could > send me (some subset of) the DB that causes the error + the query you > were trying, so I can try to reproduce the problem. > > Best, > Marijn > _______________________________________________ > postmodern-devel mailing list > postmodern-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel From marijnh at gmail.com Wed Sep 10 16:54:47 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 10 Sep 2008 18:54:47 +0200 Subject: [postmodern-devel] character encoding In-Reply-To: <30B7AC18-772B-4929-8356-9CE6E9F427DC@budzianowski.net> References: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> <7B64EEF3-FA8D-4103-8212-A918B9456B69@budzianowski.net> <30B7AC18-772B-4929-8356-9CE6E9F427DC@budzianowski.net> Message-ID: Hey Roman, There are indeed only ascii characters in that record. Anyway, I can't reproduce the problem with this information: I still don't know what your table looks like, and which encoding the database uses... if the database isn't too huge, could you try dumping it, importing it into a new database with the same encoding, testing whether the problem still occurs, and if yes, delete most of the rows, dump again, and send to me? Best, Marijn From superdog at budzianowski.net Thu Sep 11 18:18:01 2008 From: superdog at budzianowski.net (Roman Budzianowski) Date: Thu, 11 Sep 2008 11:18:01 -0700 Subject: [postmodern-devel] character encoding In-Reply-To: References: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> <7B64EEF3-FA8D-4103-8212-A918B9456B69@budzianowski.net> <30B7AC18-772B-4929-8356-9CE6E9F427DC@budzianowski.net> Message-ID: <2B812B17-F26A-4DE0-851F-4B0AE0BCE1F4@budzianowski.net> Marijn, I tried setting cl-postgres-system:*unicode* to nil and reloading. It works! I notice a non-ascii char: the registered sign. The database is in SQL_ASCII encoding, but it contains non ascii chars. Somehow psql in unicode mode deals with it, but trivial-utf-8 doesn't. So, it doesn't have anything to do with postmodern. However, maybe it would be useful to be able to set encoding mode per connection? Any other suggestions? It seems that this is a not uncommon case. Roman CL-USER> (pomo:query "SELECT item_description FROM web_source_items WHERE (id = 22225)") (("
The Aeroneb? Go Micropump Nebulizer is a fast, efficient, simple-to-use device developed for the millions of patients worldwide who require respiratory therapy in and away from home, and eliminates many of the problems associated with current methods of ...... and water, and disinfected (if desired).")) On Sep 10, 2008, at 9:54 AM, Marijn Haverbeke wrote: > Hey Roman, > > There are indeed only ascii characters in that record. Anyway, I can't > reproduce the problem with this information: I still don't know what > your table looks like, and which encoding the database uses... if the > database isn't too huge, could you try dumping it, importing it into a > new database with the same encoding, testing whether the problem still > occurs, and if yes, delete most of the rows, dump again, and send to > me? > > Best, > Marijn > _______________________________________________ > postmodern-devel mailing list > postmodern-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel From marijnh at gmail.com Thu Sep 11 18:45:43 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Thu, 11 Sep 2008 20:45:43 +0200 Subject: [postmodern-devel] character encoding In-Reply-To: <2B812B17-F26A-4DE0-851F-4B0AE0BCE1F4@budzianowski.net> References: <009989BA-5067-4776-8087-DEA36149F244@budzianowski.net> <7B64EEF3-FA8D-4103-8212-A918B9456B69@budzianowski.net> <30B7AC18-772B-4929-8356-9CE6E9F427DC@budzianowski.net> <2B812B17-F26A-4DE0-851F-4B0AE0BCE1F4@budzianowski.net> Message-ID: Hey Roman, > I notice a non-ascii char: the registered sign. The database is in SQL_ASCII > encoding, but it contains non ascii chars. Ah, I somehow missed the (R). This seems like rather awful behaviour on the part of postgres -- there's no (R) in ASCII, and sending it in latin encoding over a utf-8 encoded connection is just wrong. But yes, it would be nice if Postmodern didn't choke on it. However, I have no idea how to distinguish cases like this from genuine utf-8 without completely killing performance. > So, it doesn't have anything to do with postmodern. However, maybe it would > be useful to be able to set > encoding mode per connection? Any other suggestions? That would probably be nice. The current character-encoding approach is rather ad-hoc -- babel didn't exist yet at the time, and flexi-streams was so slow it wasn't even an option, but nowadays it should be possible to use babel to decode any format the server is likely to throw at us. Unfortunately, the current behaviour is not wrong enough to distress me to the point of making it a priority to fix it, and I have enough stuff on my hands. If anyone feels up to it, I'd be happy to review and incorporate a patch, of course. Best, Marijn From mslamin at gmail.com Sun Sep 21 00:00:20 2008 From: mslamin at gmail.com (Mark Slamin) Date: Sat, 20 Sep 2008 19:00:20 -0500 Subject: [postmodern-devel] Create-table dynamic table name Message-ID: <939680080809201700i583bcf34obb1d4484ac9fe53c@mail.gmail.com> Hi. How do I use a variable as a table name with create-table? The following always takes 'tableName' as the name rather than the value of tableName: (sql (:create-table tableName ((name :type string) (address :type string)))) :raw doesn't seem to work. Thanks Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From marijnh at gmail.com Sun Sep 21 07:45:29 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Sun, 21 Sep 2008 09:45:29 +0200 Subject: [postmodern-devel] Create-table dynamic table name In-Reply-To: <939680080809201700i583bcf34obb1d4484ac9fe53c@mail.gmail.com> References: <939680080809201700i583bcf34obb1d4484ac9fe53c@mail.gmail.com> Message-ID: Hi Mark, (sql-compile `(:create-table ,tableName ....)) should work. Best, Marijn 2008/9/21 Mark Slamin : > Hi. > How do I use a variable as a table name with create-table? > The following always takes 'tableName' as the name rather than the value of > tableName: > (sql (:create-table tableName > ((name :type string) > (address :type string)))) > :raw doesn't seem to work. > Thanks > Mark > _______________________________________________ > postmodern-devel mailing list > postmodern-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel > > From mslamin at gmail.com Tue Sep 23 22:19:21 2008 From: mslamin at gmail.com (Mark Slamin) Date: Tue, 23 Sep 2008 17:19:21 -0500 Subject: [postmodern-devel] Storing Lists Message-ID: <939680080809231519y1524790bx2c6a2040a8b53ca1@mail.gmail.com> How do I insert a list into a single field of a single record in the database? Trying to insert (A B C) yields type errors if database type is set to text or varchar. Is there another data type I should use? Does the list need to be modified somehow? I need to be able to store and retrieve a list as easily as possible. Thanks. Mark Slamin -------------- next part -------------- An HTML attachment was scrubbed... URL: From attila.lendvai at gmail.com Tue Sep 23 22:25:59 2008 From: attila.lendvai at gmail.com (Attila Lendvai) Date: Wed, 24 Sep 2008 00:25:59 +0200 Subject: [postmodern-devel] Storing Lists In-Reply-To: <939680080809231519y1524790bx2c6a2040a8b53ca1@mail.gmail.com> References: <939680080809231519y1524790bx2c6a2040a8b53ca1@mail.gmail.com> Message-ID: > How do I insert a list into a single field of a single record in the > database? use cl-serializer or cl-store to turn your lisp object into a byte array and use sql blob's to store it. -- attila From marijnh at gmail.com Wed Sep 24 06:23:39 2008 From: marijnh at gmail.com (Marijn Haverbeke) Date: Wed, 24 Sep 2008 08:23:39 +0200 Subject: [postmodern-devel] Storing Lists In-Reply-To: <939680080809231519y1524790bx2c6a2040a8b53ca1@mail.gmail.com> References: <939680080809231519y1524790bx2c6a2040a8b53ca1@mail.gmail.com> Message-ID: Hi Mark, Attila's idea works, or you could create a specific table mapping keys to multiple elements (which stores sets, not lists), or you could try Postgres' array types -- but that would require writing some extra code, since Postmodern doesn't come with readers and writers for those. Cheers, Marijn 2008/9/24 Mark Slamin : > How do I insert a list into a single field of a single record in the > database? > Trying to insert (A B C) yields type errors if database type is set to text > or varchar. > Is there another data type I should use? Does the list need to be modified > somehow? > I need to be able to store and retrieve a list as easily as possible. > Thanks. > Mark Slamin > _______________________________________________ > postmodern-devel mailing list > postmodern-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/postmodern-devel > > From spambait at CloudDancer.com Wed Sep 24 20:09:25 2008 From: spambait at CloudDancer.com (GP lisper) Date: Wed, 24 Sep 2008 13:09:25 -0700 Subject: [postmodern-devel] Re: Storing Lists References: <939680080809231519y1524790bx2c6a2040a8b53ca1@mail.gmail.com> Message-ID: On Tue, 23 Sep 2008 17:19:21 -0500, wrote: > How do I insert a list into a single field of a single record in the > database? Besides the other mentioned solutions, declare a blob field and insert it as a string. That way it is simple to read when using any SQL tool to view the database. > select tm,runs from mlb where tm="DET" order by key desc limit 1; +-----+---------------------+ | tm | runs | +-----+---------------------+ | DET | (0 0 0 0 0 0 0 0 0) | +-----+---------------------+ later, read-from-string fixes you up. -- One of the strokes of genius from McCarthy was making lists the center of the language - kt From mslamin at gmail.com Mon Sep 29 22:07:02 2008 From: mslamin at gmail.com (Mark Slamin) Date: Mon, 29 Sep 2008 17:07:02 -0500 Subject: [postmodern-devel] Re: Storing Lists In-Reply-To: References: <939680080809231519y1524790bx2c6a2040a8b53ca1@mail.gmail.com> Message-ID: <939680080809291507l1a3c4979v4fefc785219689c7@mail.gmail.com> Thanks to all for the ideas. Ends up GP Lisper's was the easiest for me to implement. I'm new to postgreSQL and Postmodern, and semi-new to Lisp, so I really appreciate the help. Thanks again, Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: