[elephant-devel] DB migration issue
quan hu
ihuquan at gmail.com
Fri Jan 2 13:15:27 UTC 2009
Hi Ian,
The 0.9.1 upgrade fix patch is attached.
I continuously run the upgrade test 3 times. All passed.
In the fix, if the DB version is <= 0.9.1, (make-instance
'bdb-indexed-btree :from-oid -3 :sc sc :indices (make-hash-table) is always
called, regardless of the indices exist before.
In my previous fix, I only call it when the indices does not exist in DB
file, but find that approach does not work. A unserialized error is
triggered when try to open the source 0.9.1 DB file for the second time.
This is caused by that the oid->schema-id has an assumption on that
instance-table is only created on a brand new DB file.
If this is not a suitable solution, please let me know.
Thanks
Quan
2009/1/2 Ian Eslick <eslick at media.mit.edu>
> pushed. -Ian
>
> On Jan 1, 2009, at 8:31 AM, quan hu wrote:
>
> > Hi Ian,
> >
> > For the pathname fix, I attached the patch.
> >
> > For the 0.9.1 upgrade fix, I got one failure and one
> > successful result.
> > I need to double check if the failure is caused by my test
> > execution step or other issue.
> > I'll send the result later.
> >
> > Thanks
> > Quan
> >
> >
> > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > I don't think we've tried the upgrade procedure in awhile, so it's
> > possible an incompatibility slipped in. It may be that all you need
> > to do is make sure that defaults are properly created for slot values
> > that are unbound during the store controller opening phase.
> >
> > Adding a case for pathname is easy enough. I take it one of the slots
> > of the pathname structure object is a function; migrating this is
> > causing the error.
> >
> > With a little work you should be able to do both of these fixes,
> > certainly the pathname fix. I'd be happy to review and commit them to
> > the repository, but unfortunately I won't have any more time to work
> > on elephant for a little while.
> >
> > Ian
> >
> > On Dec 31, 2008, at 8:37 AM, quan hu wrote:
> >
> > > Hi Ian,
> > >
> > > I downloaded the patches and have more testing today. The lock
> > > issue is gone!
> > > Thanks a lot.
> > >
> > > 1. But, another issue is found. migrate pathname fails. I test
> > > it under SBCL 1.0.21
> > > e.g.
> > > > (migrate *store-controller* #p"/tmp) => Error: Cannot
> > > migrate function objects (i.e. #<FUNCTION SB-IMPL::UNPARSE-UNIX-
> > > ENOUGH>)
> > >
> > > It is caused by that (defmethod migrate ((dst store-
> > > controller) (src structure-object) is called for this case, as
> > > pathname is taken as subtype of structure-object.
> > > > (subtypep 'pathname 'structure-object) => t t
> > > Maybe, one more migrate method is needed for pathname case.
> > >
> > > 2. I also want to know if upgrading database file from 0.9.1
> > > to unstable release is supported.
> > > When opening the DB file generated in 0.9.1 in unstable
> > > release, I get error message
> > > " The slot DB-BDB::INDICES is unbound in the object #<BDB-
> > > INDEXED-BTREE oid:-3>.
> > > [Condition of type UNBOUND-SLOT]"
> > >
> > > Thanks
> > > Quan
> > >
> > >
> > >
> > >
> > > 2008/12/31 Ian Eslick <eslick at media.mit.edu>
> > > Hi Quan,
> > >
> > > Thank you again for the information on this bug. The short story is
> > > that the way we are keeping track of transactions assumed that there
> > > were only two stores in the nested transaction stack (outer and
> > > inner). Migrating a nested object inside a with-transaction would
> > not
> > > pass the parent transaction to the backend so it wouldn't know a
> > > transaction in progress and attempted to start a new one, which
> > would
> > > block until the old one was completed.
> > >
> > > I just checked in a patch that made transaction tracking a bit more
> > > intelligent (it depends on the backends taking care of handling
> > nested
> > > transactions - which I believe they all do fine). I also checked
> > in a
> > > test which implements Quan's test case.
> > >
> > > This patch and the prior patches of the past few days pass all tests
> > > on SBCL/Mac/BDB.
> > >
> > > Ian
> > >
> > > PS - I'd love to know the current status of the elephant-unstable
> > tree
> > > with the postmodern backend.
> > >
> > >
> > > On Dec 30, 2008, at 9:34 AM, quan hu wrote:
> > >
> > > > Hello,
> > > >
> > > > After more investigation, here are some suspects:
> > > >
> > > > In the current elephant codes, the macro my-current-
> > > > transaction will check if the current transaction belongs to the
> > sc
> > > > parameter. If not, just return +NULL-CHAR+. That's exactly the
> > > > case in the following nested transaction where the store-
> > controller
> > > > is different.
> > > > The situation becomes:
> > > > Transaction 1 begins
> > > > One data is put into DB with Transaction 1
> > > > ...
> > > > Another data is put into the same DB with NULL
> > > > transaction parameter(i.e. no transaction??)
> > > >
> > > > It looks like this scenario is not supported by BDB.
> > > > I simulate above case with a C program to access BDB and
> > get
> > > > the similar failure symptom.
> > > > ...
> > > > txn1 = NULL;
> > > > ret = dbenvp1->txn_begin(dbenvp1, NULL, &txn1, 0);
> > > > if (ret != 0) {
> > > > printf ("Transaction begin failed.\n");
> > > > return 1;
> > > > }
> > > > put_record(FOO,FOO,txn1, dbp1);
> > > > printf("foo\n");
> > > > put_record(FOO,FOO,NULL, dbp1);
> > > > printf("bar\n");
> > > > txn1->commit(txn1,0);
> > > > printf("baz\n");
> > > > ...
> > > > After "foo" is printed, the C program is blocked. If I change
> > the
> > > > NULL to txn1, everything is fine.
> > > >
> > > >
> > > > Regarding to the solution, my gut feeling is that it would be
> > > > difficult to avoid this kind of nested transaction in migrate
> > > > process, because of the recursive function call. Maybe,
> > > > *transaction-stack* can be added to track the active transaction
> > > > stack. Get it pushed and poped in right place of execute-
> > > > transaction. Then, in my-current-transaction, check *transaction-
> > > > stack* to see if there is an active transaction belongs to the
> > same
> > > > store-controller. If it is found, use it. My simple test shows
> > it
> > > > works. If this is the right direction, I can do more test.
> > > >
> > > > Thanks for your help.
> > > > Quan
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > 2008/12/30 quan hu <ihuquan at gmail.com>
> > > > Hi Ian, Yarek,
> > > >
> > > > Thanks for providing the information.
> > > > I downloaded the elephant-unstable and test it with BDB4.7.
> > > > The failure symptom is the same: target store controller get
> > > > locked.
> > > >
> > > > After I make following changes in migrate.lisp; The test
> > > passed.
> > > >
> > > > (defmethod copy-btree-contents ((sc store-controller) dst
> > src)
> > > > (let (to-be-migrated)
> > > > (map-btree (lambda (key value)
> > > > (let ((newval (migrate sc
> > > value))
> > > > (newkey (migrate sc
> > > key)))
> > > > (push (list newkey
> > > > newval) to-be-migrated)))
> > > > ;;;; (setf (get-
> > value
> > > > newkey dst) newval))) ;;
> > > > src)
> > > > (loop for (k v) in to-be-migrated
> > > > do
> > > > (setf (get-value k dst) v))))
> > > >
> > > > Above debug change just avoids the nested transaction of
> > > > different store controller for the specific test case.
> > > > However, it is not the solution, as the nested transaction
> > > > still could appear in other data pattern.
> > > >
> > > > What confuses me is that the nested transaction of the same
> > > > store controller seem not bring any block.
> > > > Following test passed:
> > > > (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > > (add-to-root 'foo 'foo :sc *migrate-dst*)
> > > > (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > > (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > > >
> > > > But, the nested transaction of different store controller
> > always
> > > > results in lock.
> > > >
> > > > (with-transaction (:store-controller *migrate-dst*)
> > > >
> > > > (add-to-root 'foo 'foo :sc *migrate-dst*)
> > > > (with-transaction (:store-controller *migrate-
> > > > src*) ;; trigger database lock
> > > > (add-to-root 'bar 'bar :sc *migrate-dst*)))
> > > >
> > > > Thanks
> > > > Quan
> > > >
> > > >
> > > >
> > > > 2008/12/30 Yarek Kowalik <yarek.kowalik at gmail.com>
> > > >
> > > > Quan,
> > > >
> > > > Unstable is here:
> > > >
> > > > http://www.common-lisp.net/project/elephant/darcs/elephant-unstable
> > > >
> > > > Yarek
> > > >
> > > >
> > > > On Mon, Dec 29, 2008 at 5:37 AM, Ian Eslick <eslick at media.mit.edu>
> > > > wrote:
> > > > That would be elephant-unstable
> > > >
> > > > Sent from my iPhone
> > > >
> > > > On Dec 29, 2008, at 7:36 AM, Ian Eslick <eslick at media.mit.edu>
> > > wrote:
> > > >
> > > > > Hi Quan,
> > > > >
> > > > > Can you try the latest darcs with BDB 4.6 or 4.7? I made some
> > > > changes
> > > > > to migration, but I can't recall if they were in the latest
> > > darcs or
> > > > > not - I think the latest darcs doesn't play well with BDB 4.5 so
> > > > > please make sure you can reproduce under the above configuration
> > > if
> > > > > you can.
> > > > >
> > > > > Thank you,
> > > > > Ian
> > > > >
> > > > > On Dec 29, 2008, at 3:40 AM, quan hu wrote:
> > > > >
> > > > >> Hello,
> > > > >>
> > > > >> I run into a problem when doing the garbage collection via
> > > data
> > > > >> migration.
> > > > >> The environment is elephant 0.9.1 and BDB 4.5. I also
> > tried
> > > > >> the latest elephant in darc and get the same result.
> > > > >>
> > > > >> 1. Test case to reproduce the problem.
> > > > >>
> > > > >> (defpclass user-profile()
> > > > >> ((id :initform nil :index t)
> > > > >> (sms-inbox :initform (make-pset))))
> > > > >>
> > > > >> ;; Create a new db for test purpose
> > > > >> (setf *migrate-src* (open-store '(:bdb "/tmp/db/
> > > src/")))
> > > > >> (setf *test-obj* (make-instance 'user-profile :sc
> > > > >> *migrate-src*))
> > > > >> (insert-item 'foo (slot-value *test-obj* 'sms-
> > inbox))
> > > > >> (setf *migrate-dst* (open-store '(:bdb "/tmp/db/
> > > dst/")))
> > > > >>
> > > > >> TEST> (migrate *migrate-dst* *migrate-src*)
> > > > >> Migrating class indexes for: USER-PROFILE
> > > > >> ; => Input get blocked, not response anymore
> > > > >>
> > > > >> 2. Using "db_stat -C A" get following output:
> > > > >> 1 Lock requests not available due to conflicts,
> > > for
> > > > >> which we waited
> > > > >>
> > > > >> 3. I do some debug work and found the problem may be
> > > caused
> > > > >> by nested transaction of different store controller.
> > > > >> I can reproduce the issue with following code
> > segment
> > > > >>
> > > > >> TEST> (ensure-transaction (:store-controller *migrate-
> > > > dst*)
> > > > >> (add-to-root 'foo 'foo :sc *migrate-
> > > > dst*)
> > > > >> (ensure-transaction (:store-
> > controller
> > > > >> *migrate-src*)
> > > > >> (add-to-root 'bar 'bar :sc
> > > > *migrate-
> > > > >> dst*)))
> > > > >>
> > > > >> This kind of nested transaction scenario can appear
> > > in
> > > > >> the migrate process, because it uses source/destination store
> > > > >> controller at the same time.
> > > > >>
> > > > >> I do not understand why different store controller's
> > > > >> transaction can result in the lock conflict and want to know
> > > how to
> > > > >> fix it.
> > > > >>
> > > > >> Is there anyone meet the similar problem before?
> > > > >>
> > > > >> Thanks
> > > > >> Quan
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> _______________________________________________
> > > > >> elephant-devel site list
> > > > >> elephant-devel at common-lisp.net
> > > > >> http://common-lisp.net/mailman/listinfo/elephant-devel
> > > > >
> > > > >
> > > > > _______________________________________________
> > > > > elephant-devel site list
> > > > > elephant-devel at common-lisp.net
> > > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > > >
> > > >
> > > > _______________________________________________
> > > > elephant-devel site list
> > > > elephant-devel at common-lisp.net
> > > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> > >
> > > _______________________________________________
> > > elephant-devel site list
> > > elephant-devel at common-lisp.net
> > > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> >
> > _______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
> >
> > <migrate-pathname-
> > patch.gz>_______________________________________________
> > elephant-devel site list
> > elephant-devel at common-lisp.net
> > http://common-lisp.net/mailman/listinfo/elephant-devel
>
>
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090102/82bd0736/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: upgrade-0.9.1-fix-patch
Type: application/octet-stream
Size: 20599 bytes
Desc: not available
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090102/82bd0736/attachment.obj>
More information about the elephant-devel
mailing list