[elephant-devel] DB migration issue

Ian Eslick eslick at media.mit.edu
Thu Jan 1 20:39:17 UTC 2009


pushed.  -Ian

On Jan 1, 2009, at 8:31 AM, quan hu wrote:

> Hi Ian,
>
>      For the pathname fix, I attached the patch.
>
>      For the 0.9.1 upgrade fix,  I got one failure and one  
> successful result.
>      I need to double check if the failure is caused by my test  
> execution step or other issue.
>      I'll send the result later.
>
> Thanks
> Quan
>
>
> 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> I don't think we've tried the upgrade procedure in awhile, so it's
> possible an incompatibility slipped in.  It may be that all you need
> to do is make sure that defaults are properly created for slot values
> that are unbound during the store controller opening phase.
>
> Adding a case for pathname is easy enough.  I take it one of the slots
> of the pathname structure object is a function; migrating this is
> causing the error.
>
> With a little work you should be able to do both of these fixes,
> certainly the pathname fix.  I'd be happy to review and commit them to
> the repository, but unfortunately I won't have any more time to work
> on elephant for a little while.
>
> Ian
>
> On Dec 31, 2008, at 8:37 AM, quan hu wrote:
>
> > Hi Ian,
> >
> >      I downloaded the patches and have more testing today. The lock
> > issue is gone!
> >      Thanks a lot.
> >
> >      1. But, another issue is found. migrate pathname fails. I test
> > it under SBCL 1.0.21
> >       e.g.
> >        >  (migrate *store-controller* #p"/tmp) => Error: Cannot
> > migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-
> > ENOUGH>)
> >
> >       It is caused by that (defmethod migrate ((dst store-
> > controller) (src structure-object)  is called for this case, as
> > pathname is taken as subtype of structure-object.
> >       > (subtypep 'pathname 'structure-object) => t t
> >       Maybe, one more migrate method is needed for pathname case.
> >
> >       2. I also want to know if upgrading database file from 0.9.1
> > to unstable release is supported.
> >       When opening the DB file generated in 0.9.1 in unstable
> > release, I get error message
> >        " The slot DB-BDB::INDICES is unbound in the object #<BDB-
> > INDEXED-BTREE oid:-3>.
> >           [Condition of type UNBOUND-SLOT]"
> >
> > Thanks
> > Quan
> >
> >
> >
> >
> > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > Hi Quan,
> >
> > Thank you again for the information on this bug.  The short story is
> > that the way we are keeping track of transactions assumed that there
> > were only two stores in the nested transaction stack (outer and
> > inner).  Migrating a nested object inside a with-transaction would  
> not
> > pass the parent transaction to the backend so it wouldn't know a
> > transaction in progress and attempted to start a new one, which  
> would
> > block until the old one was completed.
> >
> > I just checked in a patch that made transaction tracking a bit more
> > intelligent (it depends on the backends taking care of handling  
> nested
> > transactions - which I believe they all do fine).  I also checked  
> in a
> > test which implements Quan's test case.
> >
> > This patch and the prior patches of the past few days pass all tests
> > on SBCL/Mac/BDB.
> >
> > Ian
> >
> > PS - I'd love to know the current status of the elephant-unstable  
> tree
> > with the postmodern backend.
> >
> >
> > On Dec 30, 2008, at 9:34 AM, quan hu wrote:
> >
> > > Hello,
> > >
> > >       After more investigation, here are some suspects:
> > >
> > >       In the current elephant codes, the macro my-current-
> > > transaction will check if the current transaction belongs to the  
> sc
> > > parameter.   If not, just return +NULL-CHAR+. That's exactly the
> > > case in the following nested transaction where the store- 
> controller
> > > is different.
> > >       The situation becomes:
> > >              Transaction 1 begins
> > >                     One data is put into DB with Transaction 1
> > >                     ...
> > >                     Another data is put into the same DB with NULL
> > > transaction parameter(i.e. no transaction??)
> > >
> > >        It looks like this scenario is not supported by BDB.
> > >        I simulate above case with a C program to access BDB and  
> get
> > > the similar failure symptom.
> > > ...
> > >   txn1 = NULL;
> > >   ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > >   if (ret != 0) {
> > >     printf ("Transaction begin failed.\n");
> > >     return 1;
> > >   }
> > >   put_record(FOO,FOO,txn1, dbp1);
> > >   printf("foo\n");
> > >   put_record(FOO,FOO,NULL, dbp1);
> > >   printf("bar\n");
> > >   txn1->commit(txn1,0);
> > >   printf("baz\n");
> > > ...
> > >  After "foo" is printed, the C program is blocked.  If I change  
> the
> > > NULL to txn1, everything is fine.
> > >
> > >
> > >  Regarding to the solution, my gut feeling is that it would be
> > > difficult to avoid this kind of nested transaction in migrate
> > > process, because of the recursive function call. Maybe,
> > > *transaction-stack*  can be added to track the active transaction
> > > stack. Get it pushed and poped in right place of execute-
> > > transaction. Then, in my-current-transaction, check *transaction-
> > > stack* to see if there is an active transaction belongs to the  
> same
> > > store-controller. If it is found, use it.   My simple test shows  
> it
> > > works. If this is the right direction, I can do more test.
> > >
> > > Thanks for your help.
> > > Quan
> > >
> > >
> > >
> > >
> > >
> > > 2008/12/30 quan hu <ihuquan at gmail.com>
> > > Hi Ian, Yarek,
> > >
> > >     Thanks for providing the information.
> > >      I downloaded the elephant-unstable and test it with BDB4.7.
> > >      The failure symptom is the same: target store controller get
> > > locked.
> > >
> > >      After I make following changes in migrate.lisp; The test
> > passed.
> > >
> > >       (defmethod copy-btree-contents ((sc store-controller) dst  
> src)
> > >             (let (to-be-migrated)
> > >                   (map-btree (lambda (key value)
> > >                                      (let ((newval (migrate sc
> > value))
> > >                                             (newkey (migrate sc
> > key)))
> > >                                            (push (list newkey
> > > newval) to-be-migrated)))
> > >                                            ;;;;    (setf (get- 
> value
> > > newkey dst) newval)))  ;;
> > >                                            src)
> > >                   (loop for (k v) in to-be-migrated
> > >                     do
> > >                     (setf (get-value k dst) v))))
> > >
> > >      Above debug change just avoids the nested transaction of
> > > different store controller for the specific test case.
> > >      However, it is not the solution, as the nested transaction
> > > still could appear in other data pattern.
> > >
> > >     What confuses me is that the nested transaction of the same
> > > store controller seem not bring any block.
> > >      Following test passed:
> > >         (with-transaction (:store-controller *migrate-dst*)
> > >
> > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > >              (with-transaction (:store-controller *migrate-dst*)
> > >
> > >                 (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > >
> > >     But, the nested transaction of different store controller  
> always
> > > results in lock.
> > >
> > >         (with-transaction (:store-controller *migrate-dst*)
> > >
> > >              (add-to-root 'foo 'foo :sc *migrate-dst*)
> > >              (with-transaction (:store-controller *migrate-
> > > src*)         ;; trigger database lock
> > >                  (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > >
> > > Thanks
> > > Quan
> > >
> > >
> > >
> > > 2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>
> > >
> > > Quan,
> > >
> > > Unstable is here:
> > >
> > >  http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> > >
> > > Yarek
> > >
> > >
> > > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu>
> > > wrote:
> > > That would be elephant-unstable
> > >
> > > Sent from my iPhone
> > >
> > > On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu>
> > wrote:
> > >
> > > > Hi Quan,
> > > >
> > > > Can you try the latest darcs with BDB 4.6 or 4.7?  I made some
> > > changes
> > > > to migration, but I can't recall if they were in the latest
> > darcs or
> > > > not - I think the latest darcs doesn't play well with BDB 4.5 so
> > > > please make sure you can reproduce under the above configuration
> > if
> > > > you can.
> > > >
> > > > Thank you,
> > > > Ian
> > > >
> > > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > > >
> > > >> Hello,
> > > >>
> > > >>     I run into a problem when doing the garbage collection via
> > data
> > > >> migration.
> > > >>     The environment is elephant 0.9.1 and BDB 4.5.  I also  
> tried
> > > >> the latest elephant in darc and get the same result.
> > > >>
> > > >>     1.  Test case to reproduce the problem.
> > > >>
> > > >>           (defpclass user-profile()
> > > >>             ((id :initform nil :index t)
> > > >>             (sms-inbox  :initform (make-pset))))
> > > >>
> > > >>            ;; Create a new db for test purpose
> > > >>           (setf *migrate-src* (open-store '(:bdb "/tmp/db/
> > src/")))
> > > >>           (setf *test-obj* (make-instance 'user-profile :sc
> > > >> *migrate-src*))
> > > >>            (insert-item 'foo (slot-value *test-obj* 'sms- 
> inbox))
> > > >>            (setf *migrate-dst* (open-store '(:bdb "/tmp/db/
> > dst/")))
> > > >>
> > > >>          TEST> (migrate *migrate-dst* *migrate-src*)
> > > >>              Migrating class indexes for: USER-PROFILE
> > > >>           ; => Input get blocked, not response anymore
> > > >>
> > > >>       2.  Using "db_stat -C A" get following output:
> > > >>             1    Lock requests not available due to conflicts,
> > for
> > > >> which we waited
> > > >>
> > > >>       3.  I do some debug work and found the problem may be
> > caused
> > > >> by nested transaction of different store controller.
> > > >>            I can reproduce the issue with following code  
> segment
> > > >>
> > > >>          TEST> (ensure-transaction (:store-controller *migrate-
> > > dst*)
> > > >>                            (add-to-root 'foo 'foo :sc *migrate-
> > > dst*)
> > > >>                            (ensure-transaction (:store- 
> controller
> > > >> *migrate-src*)
> > > >>                                 (add-to-root 'bar 'bar :sc
> > > *migrate-
> > > >> dst*)))
> > > >>
> > > >>             This kind of nested transaction scenario can appear
> > in
> > > >> the migrate process, because it uses source/destination store
> > > >> controller at the same time.
> > > >>
> > > >>            I do not understand why different store controller's
> > > >> transaction can result in the lock conflict and want to know
> > how to
> > > >> fix it.
> > > >>
> > > >>            Is there anyone meet the similar problem before?
> > > >>
> > > >> Thanks
> > > >> Quan
> > > >>
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> _______________________________________________
> > > >> elephant-devel site list
> > > >> elephant-devel at common-lisp.net
> > > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
> <migrate-pathname- 
> patch.gz>_______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel





More information about the elephant-devel mailing list