[elephant-devel] BDB Run recovery errors seen after switching to ELEPHANT-1-0-A2

Yarek Kowalik yarek.kowalik at gmail.com
Wed Mar 11 19:48:31 UTC 2009


Hi Ian,

I did some digging. Reading in the DB docs:

*DB_REGISTER*

   - Check to see if recovery needs to be performed before opening the
   database environment. (For this check to be accurate, all processes using
   the environment must specify DB_REGISTER when opening the environment.) If
   recovery needs to be performed for any reason (including the initial use of
   the DB_REGISTER flag), and DB_RECOVER is also specified, recovery will be
   performed and the open will proceed normally. If recovery needs to be
   performed and DB_RECOVER is not specified,
DB_RUNRECOVERY<http://www.oracle.com/technology/documentation/berkeley-db/db/ref/program/errorret.html#DB_RUNRECOVERY>will
be returned. If recovery does not need to be performed, the DB_RECOVER
   flag will be ignored. See Architecting Transactional Data Store
   applications<http://www.oracle.com/technology/documentation/berkeley-db/db/ref/transapp/app.html>for
more information.

I'm getting the DB_RUNRECOVERY error, which is strange since the settings
are :recover T and :register T.  So, either I'm not setting the parameters
correctly, elephant is not setting the corresponding DB_* flags correctly,
or BerkleyDB is somehow broken.

The last one is less likely since my Jan build of elephant worked, and you
indicated that the :recovery was being set to T by default then.

The first one is potentially true, but digging in whith the SLIME inspector
I can see that the controller's value of SPEC slot set as follows:

#<CONS {1007C4BCF7}>
--------------------
A proper list:
0: :BDB
1: "/home/yarek/lisp/projects/fashion-origami/data/store/"
2: :REGISTER
3: T
4: :RECOVER
5: T

I'm digging into elephant code now to see if I can see the DB_* flags being
set.  Let me know if you have any more insight.

Thanks,

Yarek




On Wed, Mar 11, 2009 at 11:42 AM, Yarek Kowalik <yarek.kowalik at gmail.com>wrote:

> - I did a fresh pull from darcs:
>
>   darcs get http://www.common-lisp.net/project/elephant/darcs/elephant-1.0
>
> - I rebuilt elephant, and weblocks/elephant
>
> - I set up my weblocks store with recover and register flags both set to T:
>
> (defstore *elephant-store* :elephant
>   :spec `(:BDB ,(namestring (merge-pathnames (make-pathname :directory
> '(:relative "data/store"))
>                          (asdf-system-directory :fashion-origami)))
>                 :register t :recover t))
>
> - Run db_recovery in the data store directory
>
> - I started the two processes (one serving port 80 the other 443).
>
> - Went to 443 first.  All is well.
>
> - Went to 80 next.  Webapp application dies (trace below).
>
> *Note:* The trace does not look much different from before.
>
> *Note2:* SLIME died when compiling elephant - got stuck on
>
> gcc -L/usr/local/BerkeleyDB.4.7/lib/ -I/usr/local/BerkeleyDB.4.7/include/
> -shared -march=x86-64 -fPIC -Wall -g -O2 -g
> /home/yarek/.sbcl/site/elephant-1.0/src/db-bdb/libberkeley-db.c -o
> /home/yarek/.sbcl/site/elephant-1.0/src/db-bdb/libberkeley-db.so -lm
>
> I recompiled from command line and got these warnings:
>
> /home/yarek/.sbcl/site/elephant-1.0/src/db-bdb/libberkeley-db.c: In
> function ‘lisp_compare2’:
> /home/yarek/.sbcl/site/elephant-1.0/src/db-bdb/libberkeley-db.c:1053:
> warning: unused variable ‘i’
> /home/yarek/.sbcl/site/elephant-1.0/src/db-bdb/libberkeley-db.c: In
> function ‘lisp_compare_key2’:
> /home/yarek/.sbcl/site/elephant-1.0/src/db-bdb/libberkeley-db.c:1147:
> warning: unused variable ‘i’
>
>
>
> Not sure how to proceed.  In order to ensure that I'm doing the right
> steps, can you please confirm:
>
> 1. Are my weblocks store settings correct?
> 2. Did I get the right repository for darcs
> 3. is manual compilation OK for libberkeley-db.so?
>
> Yarek
>
> Here is what I get as debug trace:
>
> Berkeley DB error #-30974: DB_RUNRECOVERY: Fatal error, run database
> recovery
>    [Condition of type ELEPHANT:BDB-DB-ERROR]
>
> Restarts:
>  0: [TERMINATE-THREAD] Terminate this thread (#<THREAD
> "hunchentoot-worker-2" RUNNING {1003705211}>)
>
> Backtrace:
>   0: ((LAMBDA (SWANK-BACKEND::DEBUGGER-LOOP-FN)) #<FUNCTION (LAMBDA #)
> {100296C8F9}>)
>   1: (SWANK::DEBUG-IN-EMACS #<ELEPHANT:BDB-DB-ERROR {1003081261}>)
>   2: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {1003089A39}>)
>   3: (SWANK::CALL-WITH-REDIRECTED-IO #<SWANK::CONNECTION {10036CB5E1}>
> #<CLOSURE (LAMBDA #) {1003089A59}>)
>   4: (SWANK::CALL-WITH-CONNECTION #<SWANK::CONNECTION {10036CB5E1}>
> #<CLOSURE (LAMBDA #) {1003089A39}>)
>   5: (SWANK:INVOKE-SLIME-DEBUGGER #<ELEPHANT:BDB-DB-ERROR {1003081261}>)
>   6: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
> SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {10030899F9}>)
>   7: (INVOKE-DEBUGGER #<ELEPHANT:BDB-DB-ERROR {1003081261}>)
>   8: (INVOKE-DEBUGGER #<ELEPHANT:BDB-DB-ERROR {1003081261}>)[:EXTERNAL]
>   9: ((SB-PCL::FAST-METHOD HUNCHENTOOT:MAYBE-INVOKE-DEBUGGER (T))
> #<unavailable argument> #<unavailable argument> #<ELEPHANT:BDB-DB-ERROR
> {1003081261}>)
>  10: (SIGNAL #<ELEPHANT:BDB-DB-ERROR {1003081261}>)[:EXTERNAL]
>  11: (ERROR #<ELEPHANT:BDB-DB-ERROR {1003081261}>)[:EXTERNAL]
>  12: ((FLET #:LAMBDA43) #<ELEPHANT:BDB-DB-ERROR {1003081261}>)
>  13: ((FLET #:LAMBDA43) #<ELEPHANT:BDB-DB-ERROR {1003081261}>)[:EXTERNAL]
>  14: (SIGNAL #<ELEPHANT:BDB-DB-ERROR {1003081261}>)[:EXTERNAL]
>  15: (ERROR ELEPHANT:BDB-DB-ERROR)[:EXTERNAL]
>       Locals:
>         SB-DEBUG::ARG-0 = 3
>         SB-DEBUG::ARG-1 = ELEPHANT:BDB-DB-ERROR
>  16: ((SB-PCL::FAST-METHOD ELEPHANT::EXECUTE-TRANSACTION
> (DB-BDB::BDB-STORE-CONTROLLER T)) #<unavailable argument> #<unavailable
> argument> #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>       Locals:
>         SB-DEBUG::ARG-0 = :<NOT-AVAILABLE>
>         SB-DEBUG::ARG-1 = :<NOT-AVAILABLE>
>         SB-DEBUG::ARG-2 = :<NOT-AVAILABLE>
>         SB-DEBUG::ARG-3 = :<NOT-AVAILABLE>
>         SB-DEBUG::ARG-4 = :<NOT-AVAILABLE>
>  17: (ELEPHANT::MAP-BTREE-VALUES #<unavailable lambda list>)
>       [No Locals]
>  18: (ELEPHANT::GET-DB-SCHEMAS #<unavailable lambda list>)
>       [No Locals]
>  19: (ELEPHANT:MAP-CLASS #<unavailable argument> #<unavailable
> argument>)[:EXTERNAL]
>       Locals:
>         SB-DEBUG::ARG-0 = :<NOT-AVAILABLE>
>         SB-DEBUG::ARG-1 = :<NOT-AVAILABLE>
>         SB-DEBUG::ARG-2 = :<NOT-AVAILABLE>
>  20: ((SB-PCL::FAST-METHOD WEBLOCKS:COUNT-PERSISTENT-OBJECTS
> (WEBLOCKS-ELEPHANT:ELEPHANT-STORE T)) ..)[:EXTERNAL]
>       Locals:
>         SB-DEBUG::ARG-0 = 4
>         SB-DEBUG::ARG-1 = :<NOT-AVAILABLE>
>         SB-DEBUG::ARG-2 = :<NOT-AVAILABLE>
>         SB-DEBUG::ARG-3 = #<WEBLOCKS-ELEPHANT:ELEPHANT-STORE {1002D92011}>
>         SB-DEBUG::ARG-4 = FASHION-ORIGAMI::PRODUCT-SET
>
>
>
> On Wed, Mar 11, 2009 at 10:38 AM, Ian Eslick <eslick at media.mit.edu> wrote:
>
>> BDB docs state that :register t :recover t solve the multi-process recover
>> problem.  If you have already registered a process, it will inhibit recovery
>> when a second process connects.
>>
>> I have another user with a deserialization error problem.  Try updating to
>> the latest elephant-1.0 and you should see an updated deserialization-error
>> report in the backtrace that tells you more about why there was an error.
>>
>> Ian
>>
>>
>> On Mar 11, 2009, at 1:28 PM, Yarek Kowalik wrote:
>>
>>  Ok so that did not work.
>>>
>>> I remember that last time I had to remove :recover t from OPEN-STORE
>>> method in weblocks/src/store/elephant/elephant.lisp since that was the
>>> source of problems with opening a store with two processes.  I think it
>>> forces the recovery mode at connection startup.
>>>
>>> I wonder if I should try :recover nil :register t -- doing that now.
>>>
>>> Yarek
>>>
>>> On Wed, Mar 11, 2009 at 10:06 AM, Yarek Kowalik <yarek.kowalik at gmail.com>
>>> wrote:
>>> Info:
>>>
>>> - BDB 4.7.
>>> - Ubuntu 8.04 on both 32 bit (Intel and whatever Amazon EC2 is using) and
>>> 64 bit (AMD X2 64) versions.
>>> - got the ELEPHANT-1-0-A2 via darcs
>>>
>>> I'm trying out the :register t and :recover t options now.
>>>
>>> Yarek
>>>
>>>
>>> On Tue, Mar 10, 2009 at 4:20 PM, Ian Eslick <eslick at media.mit.edu>
>>> wrote:
>>> You're using BDB 4.7, right?  What machine, os, word-width, etc?
>>>
>>> Add :register t and :recover t to the open-store keyword list.
>>>
>>> You can also try downloading from the elephant-1.0 repo but using:
>>>
>>> darcs get --tag=ELEPHANT-1-0-A2
>>> http://www.common-lisp.net/project/elephant/darcs/elephant-1.0
>>>
>>> Ian
>>>
>>>
>>>
>>>
>>> On Mar 10, 2009, at 6:57 PM, Yarek Kowalik wrote:
>>>
>>> > Hi Ian,
>>> >
>>> > Thanks for replying.
>>> >
>>> > The only difference on my end between Jan version and now is the
>>> > version of Elephant. I can switch back to the unstable version and
>>> > the current app works fine.
>>> >
>>> > Re: slots on the controler: all are set to some value - none are
>>> > unbound, but some are (see below).
>>> >
>>> > THe only reason I was upgrading was do to some other erros seen with
>>> > map-inverted-index that returned nils (though there is a way to do a
>>> > cleanup of those).
>>> >
>>> > I hope that Leslie can shed more light. Is there a way to set the
>>> > 'regsiter' flag?
>>> >
>>> > Yarek
>>> >
>>> >
>>> > #<DB-BDB::BDB-STORE-CONTROLLER {10024ADF61}>
>>> > --------------------
>>> > Class: #<STANDARD-CLASS DB-BDB::BDB-STORE-CONTROLLER>
>>> > --------------------
>>> > All Slots:
>>> > BTREES                 = #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP
>>> > #X006417A0 :TYPE (* T)> [set value] [make unbound]
>>> > CID-SEQ                = #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP
>>> > #X006439A0 :TYPE (* T)> [set value] [make unbound]
>>> > DB                     = #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP
>>> > #X00641060 :TYPE (* T)> [set value] [make unbound]
>>> > DB-VERSION             = 100 [set value] [make unbound]
>>> > DEADLOCK-DETECT-THREAD = NIL [set value] [make unbound]
>>> > DEADLOCK-PID           = NIL [set value] [make unbound]
>>> > DESERIALIZE            = ELEPHANT-SERIALIZER2::DESERIALIZE [set
>>> > value] [make unbound]
>>> > DESERIALIZE-FN         = #<FUNCTION (SB-C::&OPTIONAL-DISPATCH ..)>
>>> > [set value] [make unbound]
>>> > DUP-BTREES             = #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP
>>> > #X00641EE0 :TYPE (* T)> [set value] [make unbound]
>>> > ENVIRONMENT            = #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP
>>> > #X0063F320 :TYPE (* T)> [set value] [make unbound]
>>> > GC-MARK-LIST           = NIL [set value] [make unbound]
>>> > GC-MARK-TABLE          = NIL [set value] [make unbound]
>>> > GC-MARKING-P           = NIL [set value] [make unbound]
>>> > GC-MAX-OID             = NIL [set value] [make unbound]
>>> > INDEX-TABLE            = #<BDB-BTREE oid:-2> [set value] [make
>>> > unbound]
>>> > INDICES                = #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP
>>> > #X00642620 :TYPE (* T)> [set value] [make unbound]
>>> > INDICES-ASSOC          = #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP
>>> > #X00642D60 :TYPE (* T)> [set value] [make unbound]
>>> > INSTANCE-CACHE         = #<HASH-TABLE :TEST EQL :COUNT 8
>>> > {100271C671}> [set value] [make unbound]
>>> > INSTANCE-CACHE-LOCK    = #S(SB-THREAD:MUTEX :NAME NIL :%OWNER
>>> > NIL :STATE 0) [set value] [make unbound]
>>> > INSTANCE-CLASS-INDEX   = #<BDB-BTREE-INDEX oid:1> [set value] [make
>>> > unbound]
>>> > INSTANCE-TABLE         = #<BDB-INDEXED-BTREE oid:-3> [set value]
>>> > [make unbound]
>>> > METADATA               = #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP
>>> > #X006409B0 :TYPE (* T)> [set value] [make unbound]
>>> > OID-DB                 = #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP
>>> > #X006434E0 :TYPE (* T)> [set value] [make unbound]
>>> > OID-SEQ                = #<SB-ALIEN-INTERNALS:ALIEN-VALUE :SAP
>>> > #X00643B70 :TYPE (* T)> [set value] [make unbound]
>>> > ROOT                   = #<BDB-BTREE oid:-1> [set value] [make
>>> > unbound]
>>> > SCHEMA-CACHE           = #<HASH-TABLE :TEST EQ :COUNT 0
>>> > {100271C561}> [set value] [make unbound]
>>> > SCHEMA-CACHE-LOCK      = #S(SB-THREAD:MUTEX :NAME NIL :%OWNER
>>> > NIL :STATE 0) [set value] [make unbound]
>>> > SCHEMA-CLASSES         = NIL [set value] [make unbound]
>>> > SCHEMA-NAME-INDEX      = #<BDB-BTREE-INDEX oid:0> [set value] [make
>>> > unbound]
>>> > SCHEMA-TABLE           = #<BDB-INDEXED-BTREE oid:-4> [set value]
>>> > [make unbound]
>>> > SERIALIZE              = ELEPHANT-SERIALIZER2::SERIALIZE [set value]
>>> > [make unbound]
>>> > SERIALIZE-FN           = #<FUNCTION ELEPHANT-SERIALIZER2::SERIALIZE>
>>> > [set value] [make unbound]
>>> > SERIALIZER-VERSION     = 2 [set value] [make unbound]
>>> > SPEC                   = (:BDB "/home/yarek/lisp/projects/zzz/data/
>>> > store/" :RECOVER NIL) [set value] [make unbound]
>>> >
>>> >
>>> >
>>> >
>>> > On Tue, Mar 10, 2009 at 2:50 PM, Ian Eslick <eslick at media.mit.edu>
>>> > wrote:
>>> > Unfortunately that's not a highly informative backtrace.  Did you
>>> > upgrade to the latest, and this caused it, or did something suddenly
>>> > change that caused the January '09 version to work?
>>> >
>>> > Some possible sources of these problems:
>>> >
>>> > 1) Somehow the 'register' flag that helps support multiple processes
>>> > is causing problems; it is no longer set by default I believe.
>>> > (Leslie may know more)
>>> >
>>> > 2) The store-controller is not being opened properly.  Are all the
>>> > slots set in the controller after the second process is opened?
>>> >
>>> > Can you be more specific about what changed between January and now?
>>> >
>>> > Ian
>>> >
>>> > On Mar 10, 2009, at 5:31 PM, Yarek Kowalik wrote:
>>> >
>>> > > Hi folks,
>>> > >
>>> > > I have two processes accessing the same BDB.  One process manages
>>> > > weblocks requests on port 80, the other on port 443. In the elephant
>>> > > from last January, I was able to start, connect and use BDB from
>>> > > both processes.  Now, when I the user is redirected to port 443 and
>>> > > the process tries to retrieve data from the BDB, I get a
>>> > > DB_RUNRECOVERY error (see trace below).   This happens when the port
>>> > > 443 porcess connects for the very first time to BDB.
>>> > >
>>> > > I think I have seen this before, and I think it had to do with some
>>> > > default configuration on the controler, some argument  that forced
>>> > > the BDB into the recovery mode when process first starts up.
>>> > >
>>> > > Any idea how to resolve this? It's killing my secure connection on
>>> > > my web app - it's urgent.
>>> > >
>>> > > Thanks,
>>> > >
>>> > > Yarek
>>> > >
>>> > >
>>> > >
>>> > > Berkeley DB error #-30974: DB_RUNRECOVERY: Fatal error, run database
>>> > > recovery
>>> > >    [Condition of type ELEPHANT:BDB-DB-ERROR]
>>> > >
>>> > > Restarts:
>>> > >  0: [TERMINATE-THREAD] Terminate this thread (#<THREAD "hunchentoot-
>>> > > worker-6" RUNNING {B6CD101}>)
>>> > >
>>> > > Backtrace:
>>> > >   0: ((LAMBDA (SWANK-BACKEND::DEBUGGER-LOOP-FN)) #<FUNCTION (LAMBDA
>>> > > #) {AD4FBA5}>)
>>> > >   1: (SWANK::DEBUG-IN-EMACS #<ELEPHANT:BDB-DB-ERROR {B729BF9}>)
>>> > >   2: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
>>> > > SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {B729FED}>)
>>> > >   3: (SWANK::CALL-WITH-REDIRECTED-IO #<SWANK::CONNECTION {AF23819}>
>>> > > #<CLOSURE (LAMBDA #) {B729FFD}>)
>>> > >   4: (SWANK::CALL-WITH-CONNECTION #<SWANK::CONNECTION {AF23819}>
>>> > > #<CLOSURE (LAMBDA #) {B729FED}>)
>>> > >   5: (SWANK:INVOKE-SLIME-DEBUGGER #<ELEPHANT:BDB-DB-ERROR
>>> > {B729BF9}>)
>>> > >   6: ((LAMBDA (SWANK-BACKEND::HOOK SWANK-BACKEND::FUN)) #<FUNCTION
>>> > > SWANK:SWANK-DEBUGGER-HOOK> #<CLOSURE (LAMBDA #) {B729FCD}>)
>>> > >   7: (INVOKE-DEBUGGER #<ELEPHANT:BDB-DB-ERROR {B729BF9}>)
>>> > >   8: (INVOKE-DEBUGGER #<ELEPHANT:BDB-DB-ERROR {B729BF9}>)[:EXTERNAL]
>>> > >   9: ((SB-PCL::FAST-METHOD HUNCHENTOOT:MAYBE-INVOKE-DEBUGGER (T))
>>> > > #<unavailable argument> #<unavailable argument> #<ELEPHANT:BDB-DB-
>>> > > ERROR {B729BF9}>)
>>> > >  10: (SIGNAL #<ELEPHANT:BDB-DB-ERROR {B729BF9}>)[:EXTERNAL]
>>> > >  11: (ERROR #<ELEPHANT:BDB-DB-ERROR {B729BF9}>)[:EXTERNAL]
>>> > >  12: ((FLET #:LAMBDA43) #<ELEPHANT:BDB-DB-ERROR {B729BF9}>)
>>> > >  13: ((FLET #:LAMBDA43) #<ELEPHANT:BDB-DB-ERROR {B729BF9}>)
>>> > [:EXTERNAL]
>>> > >  14: (SIGNAL #<ELEPHANT:BDB-DB-ERROR {B729BF9}>)[:EXTERNAL]
>>> > >  15: (ERROR ELEPHANT:BDB-DB-ERROR)[:EXTERNAL]
>>> > >       Locals:
>>> > >         SB-DEBUG::ARG-0 = 3
>>> > >         SB-DEBUG::ARG-1 = ELEPHANT:BDB-DB-ERROR
>>> > >  16: ((SB-PCL::FAST-METHOD ELEPHANT:GET-VALUE (T DB-BDB::BDB-BTREE))
>>> > > #<unavailable lambda list>)
>>> > >       [No Locals]
>>> > >  17: (ELEPHANT::ENSURE-SLOT-DEF-INDEX #<unavailable argument>
>>> > > #<unavailable argument>)
>>> > >       Locals:
>>> > >         SB-DEBUG::ARG-0 = :<NOT-AVAILABLE>
>>> > >         SB-DEBUG::ARG-1 = :<NOT-AVAILABLE>
>>> > >  18: ((SB-PCL::FAST-METHOD ELEPHANT:FIND-INVERTED-INDEX
>>> > > (ELEPHANT:PERSISTENT-METACLASS T)) #<unavailable argument>
>>> > > #<unavailable argument> #<unavailable argument> #<unavailable
>>> > > argument>)[:EXTERNAL]
>>> > >       Locals:
>>> > >         SB-DEBUG::ARG-0 = :<NOT-AVAILABLE>
>>> > >         SB-DEBUG::ARG-1 = :<NOT-AVAILABLE>
>>> > >         SB-DEBUG::ARG-2 = :<NOT-AVAILABLE>
>>> > >         SB-DEBUG::ARG-3 = :<NOT-AVAILABLE>
>>> > >         SB-DEBUG::ARG-4 = :<NOT-AVAILABLE>
>>> > >  19: (ELEPHANT:MAP-INVERTED-INDEX #<unavailable argument>
>>> > > #<unavailable argument> #<unavailable argument>)[:EXTERNAL]
>>> > >       Locals:
>>> > >         SB-DEBUG::ARG-0 = :<NOT-AVAILABLE>
>>> > >         SB-DEBUG::ARG-1 = :<NOT-AVAILABLE>
>>> > >         SB-DEBUG::ARG-2 = :<NOT-AVAILABLE>
>>> > >         SB-DEBUG::ARG-3 = :<NOT-AVAILABLE>
>>> > >
>>> > >
>>> > >
>>> > > _______________________________________________
>>> > > elephant-devel site list
>>> > > elephant-devel at common-lisp.net
>>> > > http://common-lisp.net/mailman/listinfo/elephant-devel
>>> >
>>> >
>>> > _______________________________________________
>>> > elephant-devel site list
>>> > elephant-devel at common-lisp.net
>>> > http://common-lisp.net/mailman/listinfo/elephant-devel
>>> >
>>> > _______________________________________________
>>> > elephant-devel site list
>>> > elephant-devel at common-lisp.net
>>> > http://common-lisp.net/mailman/listinfo/elephant-devel
>>>
>>>
>>> _______________________________________________
>>> elephant-devel site list
>>> elephant-devel at common-lisp.net
>>> http://common-lisp.net/mailman/listinfo/elephant-devel
>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/elephant-devel/attachments/20090311/937386d9/attachment.html>


More information about the elephant-devel mailing list