From evan.monroig at gmail.com Thu Mar 13 05:19:53 2008 From: evan.monroig at gmail.com (Evan Monroig) Date: Thu, 13 Mar 2008 14:19:53 +0900 Subject: [cl-prevalence-devel] restore operation fails with linked objects Message-ID: <87bq5jxksm.fsf@monroig.net> Hi, I found a bug in cl-prevalence, where the objects that are linked are not properly reconstructed from the transaction log (restoring from a snapshot is fine). I included minimal example code in the attached file and also on lisp paste [1] for easy reference. Joe and Ma Dalton are two persons, and the slot 'PARENT of Joe is Ma. When the objects are reconstructed from the transaction log, three objects are created: Joe, Ma and another Ma that is the value of the slot 'PARENT of Joe, but not the same object as the first Ma. A look at the transaction-log.xml file shows that in the transaction for the creation of Joe, Ma is referred to as a person and the values of its slots are indicated, instead of referring to the previously created instance. Can anyone reproduce this? Any ideas to fix? I guess that the serialization doesn't know of the already existing objects. I'm working on SBCL 1.0.6 on Ubuntu 07.10 and cl-prevalence is the last released version. Thanks in advance, Evan [1] http://paste.lisp.org/display/57271 -------------- next part -------------- A non-text attachment was scrubbed... Name: cl-prevalence-bug.lisp Type: application/octet-stream Size: 2218 bytes Desc: cl-prevalence-bug.lisp URL: From ndj at hivsa.com Thu Mar 13 09:08:08 2008 From: ndj at hivsa.com (Nico de Jager) Date: Thu, 13 Mar 2008 11:08:08 +0200 Subject: [cl-prevalence-devel] restore operation fails with linked objects In-Reply-To: <87bq5jxksm.fsf@monroig.net> (Evan Monroig's message of "Thu, 13 Mar 2008 14:19:53 +0900") References: <87bq5jxksm.fsf@monroig.net> Message-ID: <87abl3xa87.fsf@davinci.hivsa.com> Evan Monroig writes: > Hi, > > I found a bug in cl-prevalence, where the objects that are linked are > not properly reconstructed from the transaction log (restoring from a > snapshot is fine). > > I included minimal example code in the attached file and also on lisp > paste [1] for easy reference. > > Joe and Ma Dalton are two persons, and the slot 'PARENT of Joe is Ma. > When the objects are reconstructed from the transaction log, three > objects are created: Joe, Ma and another Ma that is the value of the > slot 'PARENT of Joe, but not the same object as the first Ma. > > A look at the transaction-log.xml file shows that in the transaction for > the creation of Joe, Ma is referred to as a person and the values of its > slots are indicated, instead of referring to the previously created > instance. > > Can anyone reproduce this? Any ideas to fix? I guess that the > serialization doesn't know of the already existing objects. > > I'm working on SBCL 1.0.6 on Ubuntu 07.10 and cl-prevalence is the last > released version. > > Thanks in advance, > > Evan > > [1] http://paste.lisp.org/display/57271 Hi I can duplicate your reported behaviour in LispWorks Pro 5.0.2 for Linux, but I thought this was normal behaviour. When passing a compound object as parameter to a data manipulation/creation transaction function, information is saved in the transaction log to recreate such an object. I suppose code can be added to check for OBJECT-WITH-ID descendants when the transaction log is replayed to just return the same (under EQ) object (instead of reconstructing it) if such an object already exists, but in the general case (i.e. non OBJECT-WITH-ID objects) there is no way to check for this type of equality and you'll end up with duplicates. Anyone please correct me if I am wrong. So, 1) you'll either have to save Ma's ID in Joe's parent slot when using managed prevalence, or 2) don't use managed prevalence and write your own transaction functions, or wrap the managed prevalence transactions as helper-only functions in your own transactions. In this case you still pass Ma's ID to the transaction (and only Ma's ID is stored in the transaction log during the recording of the transaction in contrast to the all the info to reconstruct Ma's object). Then, inside the transaction, you can fetch Ma's object yourself and this way assign the Ma object to Joe's parent slot. I usually follow this approach. In general I don't pass compound objects to transactions, but pass IDs or the slots as parameters to construct the object inside the transaction. It's a bit tedious, though, and I am sure it can be automated with a more hefty managed prevalence implementation. HTH. Nico From scaekenberghe at common-lisp.net Thu Mar 13 09:29:26 2008 From: scaekenberghe at common-lisp.net (Sven Van Caekenberghe) Date: Thu, 13 Mar 2008 10:29:26 +0100 Subject: [cl-prevalence-devel] restore operation fails with linked objects In-Reply-To: <87abl3xa87.fsf@davinci.hivsa.com> References: <87bq5jxksm.fsf@monroig.net> <87abl3xa87.fsf@davinci.hivsa.com> Message-ID: <83611295-6E86-4143-9205-0BA32358B747@common-lisp.net> Evan, Nico, I was just looking at the test code but Nico was faster, I guess I needed more time to get back into cl-prevalence. Indeed, my first reaction was to do something like this: (defun tx-set-person-parent (system person-id parent-id) (let ((person (find-object-with-id system 'person person-id)) (parent (find-object-with-id system 'person parent-id))) (setf (parent person) parent))) (defun make-joe-and-ma-1 () (let* ((ma (make-person :firstname "Ma" :lastname "Dalton")) (joe (make-person :firstname "Joe" :lastname "Dalton" :parent ma))) (setf *ma-id* (slot-value ma 'id)) (setf *joe-id* (slot-value joe 'id)))) (defun make-joe-and-ma-2 () (let* ((ma (make-person :firstname "Ma" :lastname "Dalton")) (joe (make-person :firstname "Joe" :lastname "Dalton"))) (setf *ma-id* (slot-value ma 'id)) (setf *joe-id* (slot-value joe 'id)) (tx-set-person-parent *test-system* *joe-id* *ma-id*))) To set the link in a seperate transaction based on id's. However, (1) I haven't fully tested this (2) it seems that Evan's test code suggest that the problem is not in replaying the transaction log but in recovering from the snapshot which is different. I will continue looking into this when I have more time today... Sven On 13 Mar 2008, at 10:08, Nico de Jager wrote: > Evan Monroig writes: > >> Hi, >> >> I found a bug in cl-prevalence, where the objects that are linked are >> not properly reconstructed from the transaction log (restoring from a >> snapshot is fine). >> >> I included minimal example code in the attached file and also on lisp >> paste [1] for easy reference. >> >> Joe and Ma Dalton are two persons, and the slot 'PARENT of Joe is Ma. >> When the objects are reconstructed from the transaction log, three >> objects are created: Joe, Ma and another Ma that is the value of the >> slot 'PARENT of Joe, but not the same object as the first Ma. >> >> A look at the transaction-log.xml file shows that in the >> transaction for >> the creation of Joe, Ma is referred to as a person and the values >> of its >> slots are indicated, instead of referring to the previously created >> instance. >> >> Can anyone reproduce this? Any ideas to fix? I guess that the >> serialization doesn't know of the already existing objects. >> >> I'm working on SBCL 1.0.6 on Ubuntu 07.10 and cl-prevalence is the >> last >> released version. >> >> Thanks in advance, >> >> Evan >> >> [1] http://paste.lisp.org/display/57271 > > Hi > > I can duplicate your reported behaviour in LispWorks Pro 5.0.2 for > Linux, but I thought this was normal behaviour. When passing a > compound object as parameter to a data manipulation/creation > transaction function, information is saved in the transaction log to > recreate such an object. I suppose code can be added to check for > OBJECT-WITH-ID descendants when the transaction log is replayed to > just return the same (under EQ) object (instead of reconstructing > it) if such an object already exists, but in the general case (i.e. > non OBJECT-WITH-ID objects) there is no way to check for this type > of equality and you'll end up with duplicates. Anyone please correct > me if I am wrong. > > So, > 1) you'll either have to save Ma's ID in Joe's parent slot when > using managed prevalence, or > 2) don't use managed prevalence and write your own transaction > functions, or wrap the managed prevalence transactions as helper- > only functions in your own transactions. In this case you still pass > Ma's ID to the transaction (and only Ma's ID is stored in the > transaction log during the recording of the transaction in contrast > to the all the info to reconstruct Ma's object). Then, inside the > transaction, you can fetch Ma's object yourself and this way assign > the Ma object to Joe's parent slot. I usually follow this approach. > In general I don't pass compound objects to transactions, but pass > IDs or the slots as parameters to construct the object inside the > transaction. It's a bit tedious, though, and I am sure it can be > automated with a more hefty managed prevalence implementation. > > HTH. > Nico > _______________________________________________ > cl-prevalence-devel site list > cl-prevalence-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/cl-prevalence-devel From evan.monroig at gmail.com Thu Mar 13 09:58:16 2008 From: evan.monroig at gmail.com (Evan Monroig) Date: Thu, 13 Mar 2008 18:58:16 +0900 Subject: [cl-prevalence-devel] restore operation fails with linked objects In-Reply-To: <83611295-6E86-4143-9205-0BA32358B747@common-lisp.net> (Sven Van Caekenberghe's message of "Thu\, 13 Mar 2008 10\:29\:26 +0100") References: <87bq5jxksm.fsf@monroig.net> <87abl3xa87.fsf@davinci.hivsa.com> <83611295-6E86-4143-9205-0BA32358B747@common-lisp.net> Message-ID: <87iqzrvtc7.fsf@gmail.com> Sven Van Caekenberghe writes: > Evan, Nico, > > I was just looking at the test code but Nico was faster, I guess I > needed more time to get back into cl-prevalence. > Indeed, my first reaction was to do something like this: > > (defun tx-set-person-parent (system person-id parent-id) > (let ((person (find-object-with-id system 'person person-id)) > (parent (find-object-with-id system 'person parent-id))) > (setf (parent person) parent))) > > (defun make-joe-and-ma-2 () > (let* ((ma (make-person :firstname "Ma" :lastname "Dalton")) > (joe (make-person :firstname "Joe" :lastname "Dalton"))) > (setf *ma-id* (slot-value ma 'id)) > (setf *joe-id* (slot-value joe 'id)) > (tx-set-person-parent *test-system* *joe-id* *ma-id*))) > > To set the link in a seperate transaction based on id's. Nico, Sven, Thanks for your fast answers. It seems that I asked for the moon :). I understand that your approach to record only the IDs will work better, though I see that it can be tedious since you have to replace objects by their IDs when recording changes on them, and replace IDs by associated objects when retrieving them from the database. Actually the example was a little contrived and the slot that I have is a tree containing any number of persisted objects (that all inherit the OBJECT-WITH-ID class), but I will try using this approach and see if that can be done without too much hassle. > On 13 Mar 2008, at 10:08, Nico de Jager wrote: > >> I can duplicate your reported behaviour in LispWorks Pro 5.0.2 for >> Linux, but I thought this was normal behaviour. When passing a >> compound object as parameter to a data manipulation/creation >> transaction function, information is saved in the transaction log to >> recreate such an object. I suppose code can be added to check for >> OBJECT-WITH-ID descendants when the transaction log is replayed to >> just return the same (under EQ) object (instead of reconstructing it) >> if such an object already exists, but in the general case (i.e. non >> OBJECT-WITH-ID objects) there is no way to check for this type of >> equality and you'll end up with duplicates. Anyone please correct me >> if I am wrong. > However, (1) I haven't fully tested this (2) it seems that Evan's test > code suggest that the problem is not in replaying the transaction log > but in recovering from the snapshot which is different. I will > continue looking into this when I have more time today... (pardon me for changing the order of your writings, I hope that the flow of ideas is respected this way) As Nico wrote the behavior I'm having is not necessarily a bug, but anyway there still is a problem in that recovering from the transaction log or from the snapshot yields different results. I think that the "correct" way to go would be to check for OBJECT-WITH-ID descendants while making and/or replaying the transaction log. Thanks again, Evan From scaekenberghe at common-lisp.net Fri Mar 14 14:11:18 2008 From: scaekenberghe at common-lisp.net (Sven Van Caekenberghe) Date: Fri, 14 Mar 2008 15:11:18 +0100 Subject: [SPAM] Re: [cl-prevalence-devel] restore operation fails with linked objects In-Reply-To: <87iqzrvtc7.fsf@gmail.com> References: <87bq5jxksm.fsf@monroig.net> <87abl3xa87.fsf@davinci.hivsa.com> <83611295-6E86-4143-9205-0BA32358B747@common-lisp.net> <87iqzrvtc7.fsf@gmail.com> Message-ID: <50390E62-AC63-40E4-A850-9123D2DADB1B@common-lisp.net> Evan, On 13 Mar 2008, at 10:58, Evan Monroig wrote: > As Nico wrote the behavior I'm having is not necessarily a bug, but > anyway there still is a problem in that recovering from the > transaction > log or from the snapshot yields different results. I think that the > "correct" way to go would be to check for OBJECT-WITH-ID descendants > while making and/or replaying the transaction log. Each transaction is serialized independently from each other, using a fresh serialization-state. A snapshot is taken as a whole, correctly resolving identical objects. It would sure be a nice experiment to somehow handle object-with-id descendants (or all known objects) differently here: it somehow feels like this should work, but I am not completely sure. I took a quick look at the code and it would require some hacking into the serialization code, specifically the serialization-state (known objects). Good luck and keep us informed ! Sven PS: I annotated you lisp paste. From evan.monroig at gmail.com Sun Mar 16 08:32:28 2008 From: evan.monroig at gmail.com (Evan Monroig) Date: Sun, 16 Mar 2008 17:32:28 +0900 Subject: [SPAM] Re: [cl-prevalence-devel] restore operation fails with linked objects In-Reply-To: <50390E62-AC63-40E4-A850-9123D2DADB1B@common-lisp.net> References: <87bq5jxksm.fsf@monroig.net> <87abl3xa87.fsf@davinci.hivsa.com> <83611295-6E86-4143-9205-0BA32358B747@common-lisp.net> <87iqzrvtc7.fsf@gmail.com> <50390E62-AC63-40E4-A850-9123D2DADB1B@common-lisp.net> Message-ID: On Fri, Mar 14, 2008 at 11:11 PM, Sven Van Caekenberghe wrote: > Each transaction is serialized independently from each other, using a > fresh serialization-state. > A snapshot is taken as a whole, correctly resolving identical objects. Yes this is what I understood, so changing this would require hacking around the serialization package. > It would sure be a nice experiment to somehow handle object-with-id > descendants (or all known objects) differently here: it somehow feels > like this should work, but I am not completely sure. I took a quick > look at the code and it would require some hacking into the > serialization code, specifically the serialization-state (known > objects). Thanks for all your suggestions. I finally worked around this by transforming the objects referred to to forms that look like (list 'find-object-by-id id) where id is the integer id of the idea. When I need the information I then just call funcall and everything works :). It also somehow feels faster (I didn't benchmark), since cl-prevalence doesn't have to recurse into the objects anymore and the transaction logs are shorter. Evan