From mihai at bazon.net Sun Sep 13 10:27:31 2009 From: mihai at bazon.net (Mihai Bazon) Date: Sun, 13 Sep 2009 13:27:31 +0300 Subject: [Cl-perec-devel] Lots of questions :-) Message-ID: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> Hi folks, First off, I think you did a great job! CL-perec is very cool, but it's a pity it's poorly documented and it's hard to get started, so I thought I'd ask my questions on this list... Please note that I'm new to Lisp as well, which makes it harder for me to "use the source, Luke". I'd love to help with some howtos, once I figure out the stuff I'm after. Questions, in no particular order: 1. I'm defining my own DB package in which I :USE :CL-PEREC. This introduces some conflicts that can be mitigated by :shadowing-import-from :cl-perec :set :time. What is the recommended way, though? Would it be better if I don't :USE :cl-perec and prefix all names with "cl-perec:"? 2. I have a (defpclass* page) which shows the following warning in SBCL: ; caught STYLE-WARNING: ; defclass* for CL-PEREC:PAGE while its home package is not *package* (#) I can guess that there's a class named "page" in cl-perec? 3. OIDs seem to be randomly generated. Are they generated by cl-perec, or by Postgres itself? (I haven't use PG in many years so sorry if it's PG-related). 4. I have a Perl background and have used various DB-to-Object mappers (and even wrote my own). Most of them, in Perl, provide an easy way to "inflate/deflate" columns. For example, if a column is of type TIMESTAMP (which the DB server returns as a string), when an instance is retrieved I can make that slot return an object of type DateTime, which is more comfortable to work with. So in other words, an automatic conversion happens when a row is fetched from the DB, and the reverse conversion when it's stored into DB. Another example is storing hashed passwords in the DB. In Perl I would do: $user->password("foobar"); $user->update; ## and now $user->password is some MD5 of "foobar" Is there a way to do this with cl-perec? 5. I plan to use cl-perec with hunchentoot for creating sites. One of the things I commonly did in Perl was to use the same code (and server) for multiple websites; because they had different data, the database connection was selected at runtime, depending on the target domain name of the incoming request. I presume a way to do this in cl-perec would be to set the value of *database* accordingly on each request, but since Hunchentoot is multithreaded, would this be safe? If not, can you recommend a better way? 6. What's with the _ad, _ai, _ap, _dd, _di, _dp views? I can understand some of them, but for example _ad, _ap, _dd and _dp seem to be duplicates... 7. How do I define columns of type VARCHAR(255)? It seems wasteful to use TEXT for every string.. I think that's all. :-) Sorry for the long email and thank you for any clarifications. Cheers, -Mihai http://mihai.bazon.net/blog From attila.lendvai at gmail.com Sun Sep 13 20:53:28 2009 From: attila.lendvai at gmail.com (Attila Lendvai) Date: Sun, 13 Sep 2009 22:53:28 +0200 Subject: [Cl-perec-devel] Lots of questions :-) In-Reply-To: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> References: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> Message-ID: hi! loads of useful information is available here: http://lichteblau.blogspot.com/2009/08/cl-perec-blog-series-by-pinterface.html > 1. I'm defining my own DB package in which I :USE :CL-PEREC. ?This > ? introduces some conflicts that can be mitigated by > ? :shadowing-import-from :cl-perec :set :time. ?What is the recommended > ? way, though? ?Would it be better if I don't :USE :cl-perec and prefix > ? all names with "cl-perec:"? these are usual CL namespace (package) issues. using :shadowing-import-from is the right way, unless you want to do more magic... > 2. I have a (defpclass* page) which shows the following warning in SBCL: > > ? ; caught STYLE-WARNING: > ? ; ? defclass* for CL-PEREC:PAGE while its home package is not *package* (#) > > ? I can guess that there's a class named "page" in cl-perec? not a class, but an exported symbol. defclass* interns some symbols, and it's warning you that you are defining a class called cl-perec:page while you are in another package. this _may_ lead to problems (symbols interned into cl-perec instead of your package causing some clashes, redefinitions, etc) if you want to be tidy then you should (:shadow #:page) in your package. > 3. OIDs seem to be randomly generated. ?Are they generated by cl-perec, > ? or by Postgres itself? (I haven't use PG in many years so sorry if > ? it's PG-related). oid's encode both the class and the identity of persistent objects, that's why the oid's seem to be random. first n bits are the class, last k bits are the identity. > 4. I have a Perl background and have used various DB-to-Object mappers > ? (and even wrote my own). ?Most of them, in Perl, provide an easy way > ? to "inflate/deflate" columns. ?For example, if a column is of type > ? TIMESTAMP (which the DB server returns as a string), when an instance > ? is retrieved I can make that slot return an object of type DateTime, > ? which is more comfortable to work with. ?So in other words, an > ? automatic conversion happens when a row is fetched from the DB, and > ? the reverse conversion when it's stored into DB. > > ? Another example is storing hashed passwords in the DB. ?In Perl I > ? would do: > > ? ? ?$user->password("foobar"); > ? ? ?$user->update; > ? ? ?## and now $user->password is some MD5 of "foobar" > > ? Is there a way to do this with cl-perec? i think that's how perec works... just look at the predefined timestamp type, it returns localtime:timestamp instances. you can define your own persistent types, see the blog entries above, or look at the standard-type.lisp in perec (which should be plural, standard-types.lisp, but someone else needs to convince Levy, i've tried already... :) > 5. I plan to use cl-perec with hunchentoot for creating sites. ?One of > ? the things I commonly did in Perl was to use the same code (and > ? server) for multiple websites; because they had different data, the > ? database connection was selected at runtime, depending on the target > ? domain name of the incoming request. ?I presume a way to do this in > ? cl-perec would be to set the value of *database* accordingly on each > ? request, but since Hunchentoot is multithreaded, would this be safe? > ? If not, can you recommend a better way? you should never set these contextual variables when working with multiple threads, but bind them at a high enough point: (defmethod handle-request :around ((x some-of-my-stuff)) (let ((*database* *my-database*)) ....))) this will introduce a new separate thread-local binding of the special variable in each thread when they get to that point. btw, we will start a website Real Soon Now (weeks at most) that will demo our web architecture, including a detailed shell script that can reproduce that site locally on your machine and start it up. > 7. How do I define columns of type VARCHAR(255)? ?It seems wasteful to > ? use TEXT for every string.. (text 255) hth, and good luck with perec! -- attila From mihai at bazon.net Mon Sep 14 10:01:21 2009 From: mihai at bazon.net (Mihai Bazon) Date: Mon, 14 Sep 2009 13:01:21 +0300 Subject: [Cl-perec-devel] Lots of questions :-) In-Reply-To: References: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> Message-ID: Hi Attila, Thanks for your fast reply! Indeed, I knew about pinterface series, it helped a lot to get started. I tried defining my own type today and after some embarrassing amount of trial and error I came up with this: (cl-perec:defptype password () '(text 128)) (cl-perec::defmapping password (cl-rdbms::sql-character-varying-type :size 128) 'cl-perec:identity-reader ; unexported 'password-writer) (defun password-writer (val rdbms-values index) (setf (elt rdbms-values index) (ironclad:byte-array-to-hex-string (ironclad:digest-sequence :sha1 (babel:string-to-octets val))))) (pushnew 'password cl-perec::*canonical-types*) (pushnew 'password cl-perec::*mapped-type-precedence-list*) It works, it creates SHA1 automatically for password fields, but there's one weird side effect: now *all* columns of a TEXT type are VARCHAR(128), and they are all mangled to SHA1. :-) Could you point out what did I do wrong? Cheers, -Mihai Attila Lendvai wrote: > hi! > > loads of useful information is available here: > > http://lichteblau.blogspot.com/2009/08/cl-perec-blog-series-by-pinterface.html > > > > 1. I'm defining my own DB package in which I :USE :CL-PEREC. ?This > > ? introduces some conflicts that can be mitigated by > > ? :shadowing-import-from :cl-perec :set :time. ?What is the recommended > > ? way, though? ?Would it be better if I don't :USE :cl-perec and prefix > > ? all names with "cl-perec:"? > > > these are usual CL namespace (package) issues. using > :shadowing-import-from is the right way, unless you want to do more > magic... > > > > 2. I have a (defpclass* page) which shows the following warning in SBCL: > > > > ? ; caught STYLE-WARNING: > > ? ; ? defclass* for CL-PEREC:PAGE while its home package is not *package* (#) > > > > ? I can guess that there's a class named "page" in cl-perec? > > > not a class, but an exported symbol. defclass* interns some symbols, > and it's warning you that you are defining a class called > cl-perec:page while you are in another package. this _may_ lead to > problems (symbols interned into cl-perec instead of your package > causing some clashes, redefinitions, etc) > > if you want to be tidy then you should (:shadow #:page) in your package. > > > > 3. OIDs seem to be randomly generated. ?Are they generated by cl-perec, > > ? or by Postgres itself? (I haven't use PG in many years so sorry if > > ? it's PG-related). > > > oid's encode both the class and the identity of persistent objects, > that's why the oid's seem to be random. first n bits are the class, > last k bits are the identity. > > > > 4. I have a Perl background and have used various DB-to-Object mappers > > ? (and even wrote my own). ?Most of them, in Perl, provide an easy way > > ? to "inflate/deflate" columns. ?For example, if a column is of type > > ? TIMESTAMP (which the DB server returns as a string), when an instance > > ? is retrieved I can make that slot return an object of type DateTime, > > ? which is more comfortable to work with. ?So in other words, an > > ? automatic conversion happens when a row is fetched from the DB, and > > ? the reverse conversion when it's stored into DB. > > > > ? Another example is storing hashed passwords in the DB. ?In Perl I > > ? would do: > > > > ? ? ?$user->password("foobar"); > > ? ? ?$user->update; > > ? ? ?## and now $user->password is some MD5 of "foobar" > > > > ? Is there a way to do this with cl-perec? > > > i think that's how perec works... just look at the predefined > timestamp type, it returns localtime:timestamp instances. > > you can define your own persistent types, see the blog entries above, > or look at the standard-type.lisp in perec (which should be plural, > standard-types.lisp, but someone else needs to convince Levy, i've > tried already... :) > > > > 5. I plan to use cl-perec with hunchentoot for creating sites. ?One of > > ? the things I commonly did in Perl was to use the same code (and > > ? server) for multiple websites; because they had different data, the > > ? database connection was selected at runtime, depending on the target > > ? domain name of the incoming request. ?I presume a way to do this in > > ? cl-perec would be to set the value of *database* accordingly on each > > ? request, but since Hunchentoot is multithreaded, would this be safe? > > ? If not, can you recommend a better way? > > > you should never set these contextual variables when working with > multiple threads, but bind them at a high enough point: > > (defmethod handle-request :around ((x some-of-my-stuff)) > (let ((*database* *my-database*)) > ....))) > > this will introduce a new separate thread-local binding of the special > variable in each thread when they get to that point. > > btw, we will start a website Real Soon Now (weeks at most) that will > demo our web architecture, including a detailed shell script that can > reproduce that site locally on your machine and start it up. > > > > 7. How do I define columns of type VARCHAR(255)? ?It seems wasteful to > > ? use TEXT for every string.. > > (text 255) > > hth, and good luck with perec! > > -- > attila From attila.lendvai at gmail.com Mon Sep 14 10:40:08 2009 From: attila.lendvai at gmail.com (Attila Lendvai) Date: Mon, 14 Sep 2009 12:40:08 +0200 Subject: [Cl-perec-devel] Lots of questions :-) In-Reply-To: References: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> Message-ID: > (cl-perec:defptype password () > ?'(text 128)) > > (cl-perec::defmapping password (cl-rdbms::sql-character-varying-type :size 128) > ?'cl-perec:identity-reader ? ? ? ? ? ? ; unexported > ?'password-writer) > > (defun password-writer (val rdbms-values index) > ?(setf (elt rdbms-values index) > ? ? ? ?(ironclad:byte-array-to-hex-string > ? ? ? ? (ironclad:digest-sequence :sha1 (babel:string-to-octets val))))) i'm not sure you want to do this, though... this will apply the hash algorithm every time the slot is *written* to the db. what you want is a setf customization on the password slot that gets triggered every time the slot is *set*. but it requires a custom metaclass, etc... imho, not really worth it. i suggest to use a simple (defun update-password ...) custom function instead. *if* your password was stored with a reversible encrypting, then this solution could work if you also decrypt the slot value in the reader function of the type. btw, we have our relevant code here: http://common-lisp.net/cgi-bin/darcsweb/darcsweb.cgi/darcsweb.cgi?r=cl-dwim-cl-dwim;a=headblob;f=/model/authentication.lisp > (pushnew 'password cl-perec::*mapped-type-precedence-list*) read the docs on the above variable. you need to append your type at the end, not the beginning... hth, -- attila From pinterface at gmail.com Wed Sep 16 02:05:05 2009 From: pinterface at gmail.com (Pixel // pinterface) Date: Tue, 15 Sep 2009 20:05:05 -0600 Subject: [Cl-perec-devel] Lots of questions :-) References: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> Message-ID: Disclaimer: It's been nearly two months since I've done anything with cl-perec (too many other projects, bleh). I've done some minimal testing of what I say below, so hopefully I'm not committing any major errors, but there's no guarantee. :) "Mihai Bazon" wrote in message news:f33c75c11638fc18c81f7cb445e6924c at bazon.net... > Hi Attila, > > Thanks for your fast reply! > > Indeed, I knew about pinterface series, it helped a lot to get started. I'm glad to hear that.[0] Feel free to let me know if anything was unclear, poorly explained, or seemed to be missing. I generally know what I mean, but that doesn't mean anybody else does! :) > I tried defining my own type today and after some embarrassing amount of > trial and error I came up with this: > > (cl-perec:defptype password () > '(text 128)) (Aside: I somehow missed 'cl-perec:text entirely. Good to know about that.) > (cl-perec::defmapping password (cl-rdbms::sql-character-varying-type ...) > 'cl-perec:identity-reader ; unexported > 'password-writer) > > (defun password-writer (val rdbms-values index) > (setf (elt rdbms-values index) > (ironclad:byte-array-to-hex-string > (ironclad:digest-sequence :sha1 (babel:string-to-octets val))))) > > (pushnew 'password cl-perec::*canonical-types*) > (pushnew 'password cl-perec::*mapped-type-precedence-list*) > > It works, it creates SHA1 automatically for password fields, but there's > one weird side effect: now *all* columns of a TEXT type are > VARCHAR(128), and they are all mangled to SHA1. :-) Could you point out > what did I do wrong? The way Common Lisp types work, (subtypep 'password 'text) and (subtypep 'text 'password) are both true. Not coincidentally, one of the tests cl-perec uses to determine which reader/writer mapping to use is #'subtypep[1]. Since--according to the CL type hierarchy--your 'password type and perec's 'text types are equivalent, whichever appears in *mapped-type-precedence-list* first will determine behavior for both (and any other equivalent types). You can generally trip up #'subtypep using a satisfies test. For instance: (cl-perec:defptype password () '(and (text 128) (satisfies constantly))) Will result in[2] (subtypep 'password 'text) => t, t (subtypep 'text 'password) => nil, nil which means so long as 'password appears in *mapped-type-precedence-list* /before/ 'text, the 'password reader/writer mapping will be used for passwords, but not text. If 'text appears before 'password, 'password will be run through 'text's reader/writer mapping. HOWEVER! As Attila has already mentioned, the SQL writer is the wrong place to encrypt passwords anyway. I /think/[3] you could do something like: (defmethod (setf password-of) (new-value object) (setf (slot-value object 'password) (encrypt-password new-value))) or even (defmethod (setf password-of) (new-value object) (call-next-method (encrypt-password new-value) object)) with the caveat that you would always have to update the password using the accessor function, and never using slot-value or with-slots (unless you wanted to bypass the encryption for some reason). Attila already covered this somewhat, but it bears repeating: the SQL-reader and SQL-writer for a type must be the inverse of each other. In essence, (equivalent-p thing (sql-reader (sql-writer thing))) should be true (for whatever value of #'equivalent-p is relevant to your type). Any transformation you make in the sql-writer must be undone by the sql-reader. In general, the only time you should be defining your own SQL readers and writers is when you want to serialize an object in a funky way. E.g., if you want to convert objects of an 'e-mail class into strings of the form @ for the SQL database. Footnotes: 0. Hi! I'm the guy who is writing that particular series on cl-perec. :) 1. see #'cl-perec::mapped-type-for in persistence/type.lisp 2. If you redefine the password type within the same lisp image, be sure to remove 'password from the cl-perec::*mapped-types* hash, otherwise cl-perec won't register the change in type hierarchy.[4] 3. "think" here meaning "Warning! I haven't bothered to check!" 4. If it still doesn't register the change, restart your lisp image. perec does a lot of aggressive internal caching and gets tripped up by changes a little too easily. Hopefully that wasn't too confusing. -pix (aka pinterface) From mihai at bazon.net Wed Sep 16 06:01:17 2009 From: mihai at bazon.net (Mihai Bazon) Date: Wed, 16 Sep 2009 09:01:17 +0300 Subject: [Cl-perec-devel] Lots of questions :-) In-Reply-To: References: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> Message-ID: Hey Pixel! Thanks for your clarifications and also for your blog series. ;-) The defmethod stuff works, and you guys convinced me -- it's better than using an SQL writer. I have one more question: is there a way to use objects without "reviving instance"? The reason I'd do that is that there are objects that change very infrequently, such as Users. If I get an User object from the DB when a request comes in and I need to pass it around to various functions, it would be nice if they don't need to do (with-transaction (revive-instance user)) because that involves a (mostly useless) SELECT since the data is already there. I noticed (setf (cl-perec:persistent-p user) nil) but that feels like a dirty hack; will this save the object back in the DB if I need to change it? Cheers, -Mihai Pixel // pinterface wrote: > Disclaimer: It's been nearly two months since I've done anything with > cl-perec (too many other projects, bleh). I've done some minimal testing > of what I say below, so hopefully I'm not committing any major errors, > but there's no guarantee. :) > > "Mihai Bazon" wrote in > message news:f33c75c11638fc18c81f7cb445e6924c at bazon.net... > > Hi Attila, > > > > Thanks for your fast reply! > > > > Indeed, I knew about pinterface series, it helped a lot to get started. > > I'm glad to hear that.[0] Feel free to let me know if anything was unclear, > poorly explained, or seemed to be missing. I generally know what I mean, > but that doesn't mean anybody else does! :) > > > I tried defining my own type today and after some embarrassing amount of > > trial and error I came up with this: > > > > (cl-perec:defptype password () > > '(text 128)) > > (Aside: I somehow missed 'cl-perec:text entirely. Good to know about that.) > > > (cl-perec::defmapping password (cl-rdbms::sql-character-varying-type ...) > > 'cl-perec:identity-reader ; unexported > > 'password-writer) > > > > (defun password-writer (val rdbms-values index) > > (setf (elt rdbms-values index) > > (ironclad:byte-array-to-hex-string > > (ironclad:digest-sequence :sha1 (babel:string-to-octets val))))) > > > > (pushnew 'password cl-perec::*canonical-types*) > > (pushnew 'password cl-perec::*mapped-type-precedence-list*) > > > > It works, it creates SHA1 automatically for password fields, but there's > > one weird side effect: now *all* columns of a TEXT type are > > VARCHAR(128), and they are all mangled to SHA1. :-) Could you point out > > what did I do wrong? > > The way Common Lisp types work, (subtypep 'password 'text) and (subtypep > 'text 'password) are both true. Not coincidentally, one of the tests > cl-perec uses to determine which reader/writer mapping to use is > #'subtypep[1]. Since--according to the CL type hierarchy--your 'password > type and perec's 'text types are equivalent, whichever appears in > *mapped-type-precedence-list* first will determine behavior for both (and > any other equivalent types). > > You can generally trip up #'subtypep using a satisfies test. For instance: > (cl-perec:defptype password () > '(and (text 128) (satisfies constantly))) > Will result in[2] > (subtypep 'password 'text) => t, t > (subtypep 'text 'password) => nil, nil > which means so long as 'password appears in *mapped-type-precedence-list* > /before/ 'text, the 'password reader/writer mapping will be used for > passwords, but not text. If 'text appears before 'password, 'password will > be run through 'text's reader/writer mapping. > > HOWEVER! As Attila has already mentioned, the SQL writer is the wrong place > to encrypt passwords anyway. I /think/[3] you could do something like: > (defmethod (setf password-of) (new-value object) > (setf (slot-value object 'password) (encrypt-password new-value))) > or even > (defmethod (setf password-of) (new-value object) > (call-next-method (encrypt-password new-value) object)) > with the caveat that you would always have to update the password using the > accessor function, and never using slot-value or with-slots (unless you > wanted to bypass the encryption for some reason). > > Attila already covered this somewhat, but it bears repeating: the SQL-reader > and SQL-writer for a type must be the inverse of each other. In essence, > (equivalent-p thing (sql-reader (sql-writer thing))) > should be true (for whatever value of #'equivalent-p is relevant to your > type). Any transformation you make in the sql-writer must be undone by the > sql-reader. > > In general, the only time you should be defining your own SQL readers and > writers is when you want to serialize an object in a funky way. E.g., if > you want to convert objects of an 'e-mail class into strings of the form > @ for the SQL database. > > > Footnotes: > 0. Hi! I'm the guy who is writing that particular series on cl-perec. :) > 1. see #'cl-perec::mapped-type-for in persistence/type.lisp > 2. If you redefine the password type within the same lisp image, be sure to > remove 'password from the cl-perec::*mapped-types* hash, otherwise > cl-perec won't register the change in type hierarchy.[4] > 3. "think" here meaning "Warning! I haven't bothered to check!" > 4. If it still doesn't register the change, restart your lisp image. perec > does a lot of aggressive internal caching and gets tripped up by changes > a little too easily. > > > Hopefully that wasn't too confusing. > > -pix (aka pinterface) > > > > > > _______________________________________________ > Cl-perec-devel mailing list > Cl-perec-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/cl-perec-devel From attila.lendvai at gmail.com Wed Sep 16 08:48:20 2009 From: attila.lendvai at gmail.com (Attila Lendvai) Date: Wed, 16 Sep 2009 10:48:20 +0200 Subject: [Cl-perec-devel] Lots of questions :-) In-Reply-To: References: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> Message-ID: > I have one more question: is there a way to use objects without > "reviving instance"? ?The reason I'd do that is that there are objects > that change very infrequently, such as Users. ?If I get an User object > from the DB when a request comes in and I need to pass it around to > various functions, it would be nice if they don't need to do > (with-transaction (revive-instance user)) because that involves a > (mostly useless) SELECT since the data is already there. we do this in call-with-reloaded-authenticated-session here: http://common-lisp.net/cgi-bin/darcsweb/darcsweb.cgi/darcsweb.cgi?r=cl-dwim-cl-dwim;a=headblob;f=/meta-model/authentication.lisp the relevant line is this: (setf authenticated-session (load-instance authenticated-session :skip-existence-check #t :copy-cached-slot-values #t)) this way l-i will make a copy of the given p-o in the lisp vm, copy over any cached slot information from the source, and cache it in the transaction. as far as i remember, this is recursive, so be careful with what slots are cached, because though *-to-n associations you can end up copying a whole bunch of p-o's. also note that there's no version handling yet, so make sure you don't modify such an object, because it will write back the cached slot values, which may be long-long minutes old... -- attila From mihai at bazon.net Thu Sep 17 08:26:46 2009 From: mihai at bazon.net (Mihai Bazon) Date: Thu, 17 Sep 2009 11:26:46 +0300 Subject: [Cl-perec-devel] Lots of questions :-) In-Reply-To: References: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> Message-ID: <3864231ce6f84549e8a7823fcdb426e6@bazon.net> Hi folks, Attila Lendvai wrote: > > http://common-lisp.net/cgi-bin/darcsweb/darcsweb.cgi/darcsweb.cgi?r=cl-dwim-cl-dwim;a=headblob;f=/meta-model/authentication.lisp > > the relevant line is this: > > (setf authenticated-session (load-instance authenticated-session > :skip-existence-check #t :copy-cached-slot-values #t)) I couldn't get this to work -- I still can't access that object outside a transaction. But I'm leaving this for now. The more I dig it, the more I love it. I like inheritance, for example -- I created a base class that has an "edition" field and added a "before-committing" trigger that increments edition. Then all classes that inherit from it have a magic "version number" that gets updated for each change. It's brilliant how it works, using a single table for versions and views to map each object to its version! Great work guys! I envision that it will be a lot slower than hand-crafted SQL, but the ease of development makes it worth. I have some more questions, mostly related to optimizing performance: 1. I have a N-M relationship between Users and Groups. For this, perec creates an intermediate table, which is great because I don't have to do it myself :-) but not so great because I don't know how I can control it. When I do (push new-group (grups-of user)) perec deletes all groups of that user from the intermediate table, then re-inserts them, which makes sense because one can change the whole set of groups at once, but in the particular case of adding a new group seems overkill. I was wondering if there is a way to add a single instance. 2. Whenever I access an associated object (or a set, such as (groups-of user)) a SELECT is run. I was wondering if there is a way to have "cached" slots (in RAM, that is), at least within a transaction. 3. Is there a way to create INDEX-es on fields other than those created by defassociation or :unique slots? Cheers, -Mihai From attila.lendvai at gmail.com Thu Sep 17 08:49:41 2009 From: attila.lendvai at gmail.com (Attila Lendvai) Date: Thu, 17 Sep 2009 10:49:41 +0200 Subject: [Cl-perec-devel] Lots of questions :-) In-Reply-To: <3864231ce6f84549e8a7823fcdb426e6@bazon.net> References: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> <3864231ce6f84549e8a7823fcdb426e6@bazon.net> Message-ID: >> (setf authenticated-session (load-instance authenticated-session >> :skip-existence-check #t :copy-cached-slot-values #t)) > > I couldn't get this to work -- I still can't access that object outside > a transaction. ?But I'm leaving this for now. that statement clones a dead instance into the current transaction. silently allowing the access of instances outside a transaction can lead to many headaches, not to mention cross transaction access, so it's an error by default. changing this behavior would be trivial, but i advise you to change your requirements instead... :) > 1. I have a N-M relationship between Users and Groups. ?For this, perec > ? creates an intermediate table, which is great because I don't have to > ? do it myself :-) but not so great because I don't know how I can > ? control it. if you need full control like slots, then create the intermediate entity by hand with two associations to the two sides. that's a design decision in cl-perec, and i think a good one. > ? When I do (push new-group (grups-of user)) perec deletes all groups > ? of that user from the intermediate table, then re-inserts them, which > ? makes sense because one can change the whole set of groups at once, > ? but in the particular case of adding a new group seems overkill. ?I > ? was wondering if there is a way to add a single instance. push setf's the place, and there's no optimization for that yet. but you can use the cl-containers api functions, like insert-item (or sg like that) using the foo-of* accessors which return a cl-containers container. not sure whether the actual optimization is done or not. -- attila From levente.meszaros at gmail.com Thu Sep 17 08:54:23 2009 From: levente.meszaros at gmail.com (=?ISO-8859-1?Q?Levente_M=E9sz=E1ros?=) Date: Thu, 17 Sep 2009 10:54:23 +0200 Subject: [Cl-perec-devel] Lots of questions :-) In-Reply-To: <3864231ce6f84549e8a7823fcdb426e6@bazon.net> References: <51a88bc66b2eb5cbc8475538208d149a@bazon.net> <3864231ce6f84549e8a7823fcdb426e6@bazon.net> Message-ID: > 1. I have a N-M relationship between Users and Groups. ?For this, perec > ? creates an intermediate table, which is great because I don't have to > ? do it myself :-) but not so great because I don't know how I can > ? control it. > > ? When I do (push new-group (grups-of user)) perec deletes all groups > ? of that user from the intermediate table, then re-inserts them, which > ? makes sense because one can change the whole set of groups at once, > ? but in the particular case of adding a new group seems overkill. ?I > ? was wondering if there is a way to add a single instance. insert-item should work, see the test called test/persistence/association/m-n/collection/1 > 2. Whenever I access an associated object (or a set, such as (groups-of > ? user)) a SELECT is run. ?I was wondering if there is a way to have > ? "cached" slots (in RAM, that is), at least within a transaction. it's not yet cached automagically due to the API of inserts and removals should also be updated but you can always put things into the cache and live with them we are even doing this for a tree of instances, select them with one select and build the tree in the cache so that other functions can use the slot accessors without doing an SQL SELECT there is underlying-slot-value and underlying-slot-value-using-class to change a cached value see slot-value.lisp > 3. Is there a way to create INDEX-es on fields other than those created > ? by defassociation or :unique slots? :index #t slot option should work levy -- There's no perfectoin From mihai at bazon.net Sat Sep 19 09:59:52 2009 From: mihai at bazon.net (Mihai Bazon) Date: Sat, 19 Sep 2009 12:59:52 +0300 Subject: [Cl-perec-devel] JSON serialization Message-ID: Hi folks, I'm trying to use a custom keyword in slot definitions (:serializable) that would let a base class know whether that slot is supposed to be serialized in JSON (to implement a generic way to JSONize perec objects). Defclass-star allows arbitrary keywords, but the problem is that when defassociation is called, the values of :serializable passed in class definition are lost. What happens, I think, is that the class gets redefined but the system uses only slot keywords declared in persistent-slot-definition and subclasses. Talked to some folks on the IRC, the general opinion is to modify cl-perec, but I don't like to use modified versions of libraries I didn't wrote :). So my question is, if I implement generic support for encode-json in cl-perec, would it be acceptable to include the patch in the official repository? This support would mean the following: * keyword :serializable (or :json?) supported in slot definitions with one of the following values: :always (default), :light, :heavy, :never * (encode-json persistent-object stream) will write a JSON representation of the object that includes :always and :light slots (and the OID) * (let ((prc::*full-json* t)) (encode-json object stream)) will include the :heavy slots as well * associations will be serialized as OID (for 1-1) or [ OID, OID ... ] (for 1-n, m-n) All of the above is pretty easy to do without touching perec, but I just miss the :serializable keyword support. (or maybe I just don't know enough Lisp and there's an easy way around it? I did investigate creating my own metaclass and slot definition classes, but concluded that it's too much boilerplate code for such simple functionality) Cheers, -Mihai From mihai at bazon.net Sat Sep 19 12:11:43 2009 From: mihai at bazon.net (Mihai Bazon) Date: Sat, 19 Sep 2009 15:11:43 +0300 Subject: [Cl-perec-devel] JSON serialization In-Reply-To: References: Message-ID: <0609d212eccbcaa2a6dc4160ba975175@bazon.net> (declare (ignorable (last-email-from 'mishoo))) ;-) My mistake, it's doable without touching perec... I should take more MOP for breakfast before posting to the list. Cheers, -Mihai Mihai Bazon wrote: > Hi folks, > > I'm trying to use a custom keyword in slot definitions (:serializable) > that would let a base class know whether that slot is supposed to be > serialized in JSON (to implement a generic way to JSONize perec > objects). > > Defclass-star allows arbitrary keywords, but the problem is that when > defassociation is called, the values of :serializable passed in class > definition are lost. What happens, I think, is that the class gets > redefined but the system uses only slot keywords declared in > persistent-slot-definition and subclasses. > > Talked to some folks on the IRC, the general opinion is to modify > cl-perec, but I don't like to use modified versions of libraries I > didn't wrote :). So my question is, if I implement generic support for > encode-json in cl-perec, would it be acceptable to include the patch in > the official repository? This support would mean the following: > > * keyword :serializable (or :json?) supported in slot definitions with > one of the following values: :always (default), :light, :heavy, > :never > > * (encode-json persistent-object stream) will write a JSON > representation of the object that includes :always and :light slots > (and the OID) > > * (let ((prc::*full-json* t)) (encode-json object stream)) will include > the :heavy slots as well > > * associations will be serialized as OID (for 1-1) or [ OID, OID ... ] > (for 1-n, m-n) > > All of the above is pretty easy to do without touching perec, but I just > miss the :serializable keyword support. (or maybe I just don't know > enough Lisp and there's an easy way around it? I did investigate > creating my own metaclass and slot definition classes, but concluded > that it's too much boilerplate code for such simple functionality) > > Cheers, > -Mihai > _______________________________________________ > Cl-perec-devel mailing list > Cl-perec-devel at common-lisp.net > http://common-lisp.net/cgi-bin/mailman/listinfo/cl-perec-devel From levente.meszaros at gmail.com Sun Sep 20 08:55:06 2009 From: levente.meszaros at gmail.com (=?ISO-8859-1?Q?Levente_M=E9sz=E1ros?=) Date: Sun, 20 Sep 2009 10:55:06 +0200 Subject: [Cl-perec-devel] JSON serialization In-Reply-To: References: Message-ID: Hi, I think being able to export persistent instances into JSON is generally a good idea. It is already possible to export into binary format using cl-serializer and into XML using cl-quasi-quote. These are already implemented in cl-perec based on the mentioned libraries. As for JSON, I think it would be best implemented as an integration between cl-perec and cl-json. The function export-persistent-instances could be used as an entry point. To control what is actually exported and what is not, I don't think the suggested slot options are good enough. It is also important to note that the way of limiting what is exported from the database could be shared between the various export formats (binary, xml, json). I would probably do something more dynamic, that is store it in a context and pass that to export-persistent-instances, maybe use layered functions, so different export protocols can be reused with inheritance, or just use a lambda to tell what has to be exported. AFAICT you can't simply use slot options which are not directly supported by the slot's class. levy ps: I'm aware of your second mail -- There's no perfectoin From mihai at bazon.net Sun Sep 20 09:47:33 2009 From: mihai at bazon.net (Mihai Bazon) Date: Sun, 20 Sep 2009 12:47:33 +0300 Subject: [Cl-perec-devel] JSON serialization In-Reply-To: <50b7e5180909200121j215af6c0wf154ee20b7ca1313@mail.gmail.com> References: <0609d212eccbcaa2a6dc4160ba975175@bazon.net> <50b7e5180909200121j215af6c0wf154ee20b7ca1313@mail.gmail.com> Message-ID: (resending to the list as well) Maciej Katafiasz wrote: > On Sat, Sep 19, 2009 at 2:11 PM, Mihai Bazon wrote: > > (declare (ignorable (last-email-from 'mishoo))) ? ;-) > > > > My mistake, it's doable without touching perec... ?I should take more > > MOP for breakfast before posting to the list. > > Which shouldn't be taken to mean you shouldn't post your results regardless :) Well, I thought it's a bit lengthy and maybe not interesting for the wizards on this list and besides, I'm still not very satisfied with the outcome, but here goes. Metaclass ========= I defined a metaclass that will be used for my persistent objects instead of the default cl-perec:persistent-class. It contains a json-props slot that collects JSONizable properties as classes using this metaclass are defined. It's metaclass in turn must be computed-class:computed-class, in order to be compatible with cl-perec:persistent-class. (defparameter *json-serialization* 0) (defclass dlweb.db.metaclass (persistent-class) ((json-props :initform nil :type list :accessor json-props)) (:metaclass computed-class:computed-class)) ;; called by CLOS when a class using :metaclass dlweb.db.metaclass is ;; (re)defined (defmethod shared-initialize :after ((class dlweb.db.metaclass) slot-names &rest initargs &key direct-slots) (declare (ignore initargs slot-names)) ;; (format t "~A~%~A~%~%" slot-names initargs) (loop :for slot :in direct-slots :for name = (getf slot :name) :for json = (getf slot :json 0) :when (and json (not (getf slot :instance))) :do (setf (getf (json-props class) name) json))) Shared-initialize walks through direct-slots (unfortunately I couldn't figure out how to cover inherited slots as well) and looks for a "json" property. If not found, the default value is zero (always serializable). Unless this property is nil, the slot and the value of :json is collected in json-props. [ this property specifies the "weight" of the slot for JSON encoding. The default (zero) means very light and is always serialized, while heavier props are controlled by *json-serialization*; see encode-json below ] In order to allow a :json keyword on slot definitions without writing loads of code to define my own slot classes etc. I had to do the following: (pushnew :json defclass-star:*allowed-slot-definition-properties*) Unfortunately, this property can only be accessed in shared-initialize -- it doesn't really exist in slot objects. But it does the job for now. Defining serializable objects ============================= I defined some wrappers around defpclass* and defassociation*: (defmacro my-defpclass (name &body args) `(pushnew (defpclass* ,name , at args (:metaclass dlweb.db.metaclass)) *dlweb-persistent-classes*)) (defmacro my-def-association (&body args) `(pushnew (prc::find-association (defassociation* , at args)) *dlweb-persistent-classes*)) [ Note one small inconsistency here, defpclass* returns a class object, while defassociation* returns an association name instead of the object itself, so I had to call find-association ] They automatically add my :metaclass to class definitions and push the newly created classes in a *dlweb-persistent-classes* variable, which I'll use next to deploy the schema into the database: (defparameter *dlweb-persistent-classes* nil) (defun db-deploy-schema () (mapc #'prc::export-to-rdbms (reverse *dlweb-persistent-classes*))) Base class for serializable objects =================================== I'm using the following class as the base class for all serializable objects. It does two things: 1. maintain a version number for all objects (edition-of object). It is incremented automagically when an object is saved to DB using a "before-committing-instance" trigger 2. provide an encode-json method that walks through the json-props list and builds a JSON representation of the object. (my-defpclass db-versioned-object () ((edition 0 :type integer-32))) (defmethod before-committing-instance ((transaction transaction-mixin) (-self- db-versioned-object) event) (declare (ignore transaction)) (when (eq event :modified) (incf (edition-of -self-))) (call-next-method)) (defun find-slot (class name) (when (symbolp class) (setq class (find-class class))) (find-if #'(lambda(sl) (eq (slot-definition-name sl) name)) (class-slots class))) ;; maximum weight allowed for serialization (defparameter *json-serialization* 0) ;; this one is ugly, should clean it up (defmethod encode-json ((object db-versioned-object) &optional stream) (let ((props (loop :for (name weight) :on (json-props (class-of object)) :by #'cddr :when (<= weight *json-serialization*) :collect (cons name (let* ((slot (find-slot (class-of object) name)) (val (slot-value object name))) (cond ((typep slot 'persistent-association-end-effective-slot-definition) (if (listp val) (mapcar #'oid-of val) (oid-of val))) (t val))))))) ;; note that because inherited slots are not covered by the ;; metaclass, edition must be added manually. This sucks. (push (cons 'edition (edition-of object)) props) (push (cons 'id (oid-of object)) props) (encode-json props stream))) ;; you might also find the following useful: it encodes timestamp ;; objects as number of milliseconds since Jan 1 1970 00:00:00 UTC, ;; which I find preferable in JS as I can say new Date(milliseconds) (defmethod encode-json ((value timestamp) &optional stream) (encode-json (+ (* 1000 (timestamp-to-unix value)) (timestamp-millisecond value)) stream)) Example usage ============= (my-defpclass db-user (db-versioned-object) ((create-time (transaction-timestamp) :type timestamp :index t) (last-seen nil :type (or null timestamp) :index t) (userid :type (text 32) :unique t) (name nil :type (or null (text 64)) :index t) (email :type (text 128) :unique t) (password nil :type (or nil (text 128)) :json nil))) ;; password not serialized (my-defpclass db-group (db-versioned-object) ((name :type (text 32) :unique t) (description :type (text 255)))) (my-def-association ((:class db-user :slot groups :type (prc:set db-group) :json 5) ;; 5 is "heavy" (:class db-group :slot users :type (prc:set db-user) :json nil)) ;; don't serialize users (:cache t)) (db-deploy-schema) (defvar groups (with-transaction (list (make-instance 'db-group :name "admin" :description "Power is not for everyone, so take care!") (make-instance 'db-group :name "editor" :description "Users allowed to edit pages") (make-instance 'db-group :name "programmer" :description "Users allowed to mess with internals")))) (defvar u (with-transaction (make-instance 'db-user :name "Mihai Bazon" :userid "mishoo" :email "mihai at bazon.net" :password "foo" :groups groups))) (with-transaction (revive-instance u) (encode-json u)) ;; {"id":325330,"edition":0,"email":"mihai at bazon.net","name":"Mihai ;; Bazon","userid":"mishoo","lastSeen":null,"createTime":1253439288138} (with-transaction (revive-instance u) (encode-json (groups-of u))) ;; [{"id":69012,"edition":0,"description":"Power is not for everyone, ;; so take ;; care!","name":"admin"},{"id":134548,"edition":0,"description":"Users ;; allowed to edit ;; pages","name":"editor"},{"id":200084,"edition":0,"description":"Users ;; allowed to mess with internals","name":"programmer"}] ;; the following includes the "groups" array because we increase the ;; maximum weight: (let ((*json-serialization* 5)) (with-transaction (revive-instance u) (encode-json u))) ;; {"id":325330,"edition":0,"comments":null,"pages":null, ;; "groups":[69012,134548,200084],"email":"mihai at bazon.net","name":"Mihai Bazon", ;; "userid":"mishoo","lastSeen":null,"createTime":1253439288138} That's it ========= Hope someone will find it useful. I'm still learning, if I made any blatant mistake or if anything I wrote can be written nicer/cleaner, I'd appreciate any hints. ;-) Cheers, -Mihai