From vsedach at gmail.com Sun Sep 3 04:49:33 2006 From: vsedach at gmail.com (Vladimir Sedach) Date: Sat, 2 Sep 2006 22:49:33 -0600 Subject: [elephant-devel] Berkeley DB 4.4.20 Patch Message-ID: Hi everybody, Unless the BDB people are lying in their Changelog, the only thing that changed between 4.3 and 4.4 as far as Elephant should be concerned are the constants in db.h. Diff information provided below. I don't use Elephant for anything right now (but I am planning to in the future, and what better way to get familiar than fixing things like this?), so the test suite is the only thing I have to go by. For both BDB 4.3 and 4.4, do-backend-tests passes everything but PCURSOR2 and INDEXING-REDEF-CLASS. Should I be worried? Vladimir cd /home/viper/programming/CL/elephant-work/src/db-bdb/ diff -c /home/viper/programming/CL/elephant/src/db-bdb/sleepycat.lisp /home/viper/programming/CL/elephant-work/src/db-bdb/sleepycat.lisp *** /home/viper/programming/CL/elephant/src/db-bdb/sleepycat.lisp 2006-04-29 19:03:49.000000000 -0600 --- /home/viper/programming/CL/elephant-work/src/db-bdb/sleepycat.lisp 2006-09-02 22:05:23.000000000 -0600 *************** *** 119,151 **** (defconstant DB-UNKNOWN 5) (defconstant DB_AUTO_COMMIT #x1000000) ! (defconstant DB_JOINENV #x0040000) ! (defconstant DB_INIT_CDB #x0001000) ! (defconstant DB_INIT_LOCK #x0002000) ! (defconstant DB_INIT_LOG #x0004000) ! (defconstant DB_INIT_MPOOL #x0008000) ! (defconstant DB_INIT_REP #x0010000) ! (defconstant DB_INIT_TXN #x0020000) (defconstant DB_RECOVER #x0000020) (defconstant DB_RECOVER_FATAL #x0200000) (defconstant DB_LOCKDOWN #x0080000) (defconstant DB_PRIVATE #x0100000) ! (defconstant DB_SYSTEM_MEM #x0400000) (defconstant DB_THREAD #x0000040) (defconstant DB_FORCE #x0000004) ! (defconstant DB_DEGREE_2 #x2000000) (defconstant DB_DIRTY_READ #x4000000) (defconstant DB_CREATE #x0000001) ! (defconstant DB_EXCL #x0001000) (defconstant DB_NOMMAP #x0000008) (defconstant DB_RDONLY #x0000010) (defconstant DB_TRUNCATE #x0000080) (defconstant DB_TXN_NOSYNC #x0000100) ! (defconstant DB_TXN_NOWAIT #x0001000) ! (defconstant DB_TXN_SYNC #x0002000) (defconstant DB_LOCK_NOWAIT #x002) ! (defconstant DB_DUP #x0000002) ! (defconstant DB_DUPSORT #x0000004) (defconstant DB_CURRENT 7) (defconstant DB_FIRST 9) --- 119,152 ---- (defconstant DB-UNKNOWN 5) (defconstant DB_AUTO_COMMIT #x1000000) ! (defconstant DB_JOINENV #x0000000) ! (defconstant DB_INIT_CDB #x0002000) ! (defconstant DB_INIT_LOCK #x0004000) ! (defconstant DB_INIT_LOG #x0008000) ! (defconstant DB_INIT_MPOOL #x0010000) ! (defconstant DB_INIT_REP #x0020000) ! (defconstant DB_INIT_TXN #x0040000) (defconstant DB_RECOVER #x0000020) (defconstant DB_RECOVER_FATAL #x0200000) (defconstant DB_LOCKDOWN #x0080000) (defconstant DB_PRIVATE #x0100000) ! (defconstant DB_SYSTEM_MEM #x0800000) (defconstant DB_THREAD #x0000040) (defconstant DB_FORCE #x0000004) ! (defconstant DB_DEGREE_2 #x2000000) ;; DEPRECATED, now called DB_READ_COMMITTED ! (defconstant DB_READ_COMMITTED #x2000000) (defconstant DB_DIRTY_READ #x4000000) (defconstant DB_CREATE #x0000001) ! (defconstant DB_EXCL #x0002000) (defconstant DB_NOMMAP #x0000008) (defconstant DB_RDONLY #x0000010) (defconstant DB_TRUNCATE #x0000080) (defconstant DB_TXN_NOSYNC #x0000100) ! (defconstant DB_TXN_NOWAIT #x0002000) ! (defconstant DB_TXN_SYNC #x0004000) (defconstant DB_LOCK_NOWAIT #x002) ! (defconstant DB_DUP #x0004000) ! (defconstant DB_DUPSORT #x0008000) (defconstant DB_CURRENT 7) (defconstant DB_FIRST 9) Diff finished at Sat Sep 2 22:26:08 From read at robertlread.net Sun Sep 3 05:23:34 2006 From: read at robertlread.net (Robert L. Read) Date: Sun, 03 Sep 2006 00:23:34 -0500 Subject: [elephant-devel] Berkeley DB 4.4.20 Patch In-Reply-To: References: Message-ID: <1157261015.15135.414.camel@localhost.localdomain> Wow! Thanks for this patch. I will try to integrate it into the repository early next week. "Should you be worried?" -- If all but those two test pass, you probably have a usable system for most purposes, and it will take you a long time to trip over whatever bugs causes those two failures. But, we should try to get to the bottom of those failures; in particular I would like to know if they are related to BDB 4.3 vs. BDB 4.4, or if they simply represent a bug that your LISP version exposes. Either are possible. The tests were all green on all of the systems that Ian and I could easily test on at the time of the release; I wrote those in this group, and asked for other test results, but got very few responses. What architecture, OS, and LISP version are you using? Do we need to be both 4.3 and 4.4 compatible? Can I just leave the two files in the repository and document when to use the 4.3 version? On Sat, 2006-09-02 at 22:49 -0600, Vladimir Sedach wrote: > Hi everybody, > > Unless the BDB people are lying in their Changelog, the only thing > that changed between 4.3 and 4.4 as far as Elephant should be > concerned are the constants in db.h. Diff information provided below. > I don't use Elephant for anything right now (but I am planning to in > the future, and what better way to get familiar than fixing things > like this?), so the test suite is the only thing I have to go by. For > both BDB 4.3 and 4.4, do-backend-tests passes everything but PCURSOR2 > and INDEXING-REDEF-CLASS. Should I be worried? > > Vladimir > > cd /home/viper/programming/CL/elephant-work/src/db-bdb/ > diff -c /home/viper/programming/CL/elephant/src/db-bdb/sleepycat.lisp > /home/viper/programming/CL/elephant-work/src/db-bdb/sleepycat.lisp > *** /home/viper/programming/CL/elephant/src/db-bdb/sleepycat.lisp 2006-04-29 > 19:03:49.000000000 -0600 > --- /home/viper/programming/CL/elephant-work/src/db-bdb/sleepycat.lisp 2006-09-02 > 22:05:23.000000000 -0600 > *************** > *** 119,151 **** > (defconstant DB-UNKNOWN 5) > > (defconstant DB_AUTO_COMMIT #x1000000) > ! (defconstant DB_JOINENV #x0040000) > ! (defconstant DB_INIT_CDB #x0001000) > ! (defconstant DB_INIT_LOCK #x0002000) > ! (defconstant DB_INIT_LOG #x0004000) > ! (defconstant DB_INIT_MPOOL #x0008000) > ! (defconstant DB_INIT_REP #x0010000) > ! (defconstant DB_INIT_TXN #x0020000) > (defconstant DB_RECOVER #x0000020) > (defconstant DB_RECOVER_FATAL #x0200000) > (defconstant DB_LOCKDOWN #x0080000) > (defconstant DB_PRIVATE #x0100000) > ! (defconstant DB_SYSTEM_MEM #x0400000) > (defconstant DB_THREAD #x0000040) > (defconstant DB_FORCE #x0000004) > ! (defconstant DB_DEGREE_2 #x2000000) > (defconstant DB_DIRTY_READ #x4000000) > (defconstant DB_CREATE #x0000001) > ! (defconstant DB_EXCL #x0001000) > (defconstant DB_NOMMAP #x0000008) > (defconstant DB_RDONLY #x0000010) > (defconstant DB_TRUNCATE #x0000080) > (defconstant DB_TXN_NOSYNC #x0000100) > ! (defconstant DB_TXN_NOWAIT #x0001000) > ! (defconstant DB_TXN_SYNC #x0002000) > (defconstant DB_LOCK_NOWAIT #x002) > ! (defconstant DB_DUP #x0000002) > ! (defconstant DB_DUPSORT #x0000004) > > (defconstant DB_CURRENT 7) > (defconstant DB_FIRST 9) > --- 119,152 ---- > (defconstant DB-UNKNOWN 5) > > (defconstant DB_AUTO_COMMIT #x1000000) > ! (defconstant DB_JOINENV #x0000000) > ! (defconstant DB_INIT_CDB #x0002000) > ! (defconstant DB_INIT_LOCK #x0004000) > ! (defconstant DB_INIT_LOG #x0008000) > ! (defconstant DB_INIT_MPOOL #x0010000) > ! (defconstant DB_INIT_REP #x0020000) > ! (defconstant DB_INIT_TXN #x0040000) > (defconstant DB_RECOVER #x0000020) > (defconstant DB_RECOVER_FATAL #x0200000) > (defconstant DB_LOCKDOWN #x0080000) > (defconstant DB_PRIVATE #x0100000) > ! (defconstant DB_SYSTEM_MEM #x0800000) > (defconstant DB_THREAD #x0000040) > (defconstant DB_FORCE #x0000004) > ! (defconstant DB_DEGREE_2 #x2000000) ;; DEPRECATED, now called > DB_READ_COMMITTED > ! (defconstant DB_READ_COMMITTED #x2000000) > (defconstant DB_DIRTY_READ #x4000000) > (defconstant DB_CREATE #x0000001) > ! (defconstant DB_EXCL #x0002000) > (defconstant DB_NOMMAP #x0000008) > (defconstant DB_RDONLY #x0000010) > (defconstant DB_TRUNCATE #x0000080) > (defconstant DB_TXN_NOSYNC #x0000100) > ! (defconstant DB_TXN_NOWAIT #x0002000) > ! (defconstant DB_TXN_SYNC #x0004000) > (defconstant DB_LOCK_NOWAIT #x002) > ! (defconstant DB_DUP #x0004000) > ! (defconstant DB_DUPSORT #x0008000) > > (defconstant DB_CURRENT 7) > (defconstant DB_FIRST 9) > > Diff finished at Sat Sep 2 22:26:08 > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Sun Sep 3 06:55:32 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sun, 03 Sep 2006 02:55:32 -0400 Subject: [elephant-devel] Re: DB 4.4.20 Patch Message-ID: <44FA7C64.4070603@csail.mit.edu> Way to go! Did you test against the current HEAD or the 0.6.0 release branch? All tests passed on 0.6.0 so if you are working from that release you should be worried, but it's possible that not all tests pass the current HEAD. Also, did they pass before and after your fix? In any event, I presume this change to the lisp code will not be backward compatible to work with the 4.3 libraries? I'll see about rolling this change into HEAD tomorrow. HEAD is slowly moving towards an 0.6.1 release One of the nice advantages of 4.4 over 4.3 is that the system can compact btree storage without having to do a dump and reload. There are a couple of other small projects to tackle for 0.6.1 if you'd like to get more familiar with the system! I've been making some strides towards better thread safety this week. I hope to check in those changes shortly. Cheers, Ian From eslick at csail.mit.edu Sun Sep 3 05:29:07 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sun, 03 Sep 2006 01:29:07 -0400 Subject: [elephant-devel] Berkeley DB 4.4.20 Patch In-Reply-To: References: Message-ID: <44FA6823.1080401@csail.mit.edu> Way to go! Did you test against the current HEAD or the 0.6.0 release branch? All tests passed on 0.6.0 so if you are working from that release you should be worried, but it's possible that not all tests pass the current HEAD. Also, did they pass before and after your fix? In any event, I presume this change to the lisp code will not be backward compatible to work with the 4.3 libraries? I'll see about rolling this change into HEAD tomorrow. HEAD is slowly moving towards an 0.6.1 release One of the nice advantages of 4.4 over 4.3 is that the system can compact btree storage without having to do a dump and reload. There are a couple of other small projects to tackle for 0.6.1 if you'd like to get more familiar with the system! I've been making some strides towards better thread safety this week. I hope to check in those changes shortly. Cheers, Ian Vladimir Sedach wrote: > Hi everybody, > > Unless the BDB people are lying in their Changelog, the only thing > that changed between 4.3 and 4.4 as far as Elephant should be > concerned are the constants in db.h. Diff information provided below. > I don't use Elephant for anything right now (but I am planning to in > the future, and what better way to get familiar than fixing things > like this?), so the test suite is the only thing I have to go by. For > both BDB 4.3 and 4.4, do-backend-tests passes everything but PCURSOR2 > and INDEXING-REDEF-CLASS. Should I be worried? > > Vladimir > > cd /home/viper/programming/CL/elephant-work/src/db-bdb/ > diff -c /home/viper/programming/CL/elephant/src/db-bdb/sleepycat.lisp > /home/viper/programming/CL/elephant-work/src/db-bdb/sleepycat.lisp > *** /home/viper/programming/CL/elephant/src/db-bdb/sleepycat.lisp > 2006-04-29 > 19:03:49.000000000 -0600 > --- > /home/viper/programming/CL/elephant-work/src/db-bdb/sleepycat.lisp > 2006-09-02 > 22:05:23.000000000 -0600 > *************** > *** 119,151 **** > (defconstant DB-UNKNOWN 5) > > (defconstant DB_AUTO_COMMIT #x1000000) > ! (defconstant DB_JOINENV #x0040000) > ! (defconstant DB_INIT_CDB #x0001000) > ! (defconstant DB_INIT_LOCK #x0002000) > ! (defconstant DB_INIT_LOG #x0004000) > ! (defconstant DB_INIT_MPOOL #x0008000) > ! (defconstant DB_INIT_REP #x0010000) > ! (defconstant DB_INIT_TXN #x0020000) > (defconstant DB_RECOVER #x0000020) > (defconstant DB_RECOVER_FATAL #x0200000) > (defconstant DB_LOCKDOWN #x0080000) > (defconstant DB_PRIVATE #x0100000) > ! (defconstant DB_SYSTEM_MEM #x0400000) > (defconstant DB_THREAD #x0000040) > (defconstant DB_FORCE #x0000004) > ! (defconstant DB_DEGREE_2 #x2000000) > (defconstant DB_DIRTY_READ #x4000000) > (defconstant DB_CREATE #x0000001) > ! (defconstant DB_EXCL #x0001000) > (defconstant DB_NOMMAP #x0000008) > (defconstant DB_RDONLY #x0000010) > (defconstant DB_TRUNCATE #x0000080) > (defconstant DB_TXN_NOSYNC #x0000100) > ! (defconstant DB_TXN_NOWAIT #x0001000) > ! (defconstant DB_TXN_SYNC #x0002000) > (defconstant DB_LOCK_NOWAIT #x002) > ! (defconstant DB_DUP #x0000002) > ! (defconstant DB_DUPSORT #x0000004) > > (defconstant DB_CURRENT 7) > (defconstant DB_FIRST 9) > --- 119,152 ---- > (defconstant DB-UNKNOWN 5) > > (defconstant DB_AUTO_COMMIT #x1000000) > ! (defconstant DB_JOINENV #x0000000) > ! (defconstant DB_INIT_CDB #x0002000) > ! (defconstant DB_INIT_LOCK #x0004000) > ! (defconstant DB_INIT_LOG #x0008000) > ! (defconstant DB_INIT_MPOOL #x0010000) > ! (defconstant DB_INIT_REP #x0020000) > ! (defconstant DB_INIT_TXN #x0040000) > (defconstant DB_RECOVER #x0000020) > (defconstant DB_RECOVER_FATAL #x0200000) > (defconstant DB_LOCKDOWN #x0080000) > (defconstant DB_PRIVATE #x0100000) > ! (defconstant DB_SYSTEM_MEM #x0800000) > (defconstant DB_THREAD #x0000040) > (defconstant DB_FORCE #x0000004) > ! (defconstant DB_DEGREE_2 #x2000000) ;; DEPRECATED, now called > DB_READ_COMMITTED > ! (defconstant DB_READ_COMMITTED #x2000000) > (defconstant DB_DIRTY_READ #x4000000) > (defconstant DB_CREATE #x0000001) > ! (defconstant DB_EXCL #x0002000) > (defconstant DB_NOMMAP #x0000008) > (defconstant DB_RDONLY #x0000010) > (defconstant DB_TRUNCATE #x0000080) > (defconstant DB_TXN_NOSYNC #x0000100) > ! (defconstant DB_TXN_NOWAIT #x0002000) > ! (defconstant DB_TXN_SYNC #x0004000) > (defconstant DB_LOCK_NOWAIT #x002) > ! (defconstant DB_DUP #x0004000) > ! (defconstant DB_DUPSORT #x0008000) > > (defconstant DB_CURRENT 7) > (defconstant DB_FIRST 9) > > Diff finished at Sat Sep 2 22:26:08 > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From vsedach at gmail.com Sun Sep 3 17:48:40 2006 From: vsedach at gmail.com (Vladimir Sedach) Date: Sun, 3 Sep 2006 11:48:40 -0600 Subject: [elephant-devel] Berkeley DB 4.4.20 Patch In-Reply-To: <44FA6823.1080401@csail.mit.edu> References: <44FA6823.1080401@csail.mit.edu> Message-ID: Hello, I was working against HEAD circa late last week. I guess I should have realized there might be some broken code in the repository. :) The same tests passed before and after, so at least it doesn't look like I broke anything new. Since only the flag constants changed as far as the interface to BDB is concerned, it wouldn't be a big problem to keep both 4.3 and 4.4 code around. However, I don't see any reason to do it. 4.4 fixes a ton of bugs, and migrating to 4.4 (not that it would be a problem for Elephant, as judging from this mailing list traffic no one uses the BDB backend for anything important :)) is fully supported by BDB itself. I'll take a look around the TODO and hack on things as I have time. As far as my platform goes, I'm running SBCL 0.9.16 on Linux on Pentium IV and also sometimes (but not this time) AMD64 (with the same 32-bit binaries). Vladimir From eslick at csail.mit.edu Mon Sep 4 03:49:38 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sun, 03 Sep 2006 23:49:38 -0400 Subject: [elephant-devel] Berkeley DB 4.4.20 Patch In-Reply-To: References: Message-ID: <44FBA252.5030209@csail.mit.edu> Vladimir, I applied your changes, cleaned up some other changes for a check in. I found the same errors as you on Mac OS using Allegro. Digging deeper into this it appears there are genuine issues cropping up in these two tests. It appears that a pcursor-next operation returns nil instead of t and prematurely terminates the walk of an index. This may be related to the advertised change in the auto-commit behavior. Auto-commit used to be a default that could be overridden within a transaction. The new behavior is global so either you always auto-commit or you have to wrap transactions around everything. BDB users relying on the old interleaving behavior may be heavily effected by the 4.4 upgrade. I'll continue to look into this tomorrow so I better understand the implications. Regards, Ian Vladimir Sedach wrote: > Hi everybody, > > Unless the BDB people are lying in their Changelog, the only thing > that changed between 4.3 and 4.4 as far as Elephant should be > concerned are the constants in db.h. Diff information provided below. > I don't use Elephant for anything right now (but I am planning to in > the future, and what better way to get familiar than fixing things > like this?), so the test suite is the only thing I have to go by. For > both BDB 4.3 and 4.4, do-backend-tests passes everything but PCURSOR2 > and INDEXING-REDEF-CLASS. Should I be worried? > > Vladimir > > cd /home/viper/programming/CL/elephant-work/src/db-bdb/ > diff -c /home/viper/programming/CL/elephant/src/db-bdb/sleepycat.lisp > /home/viper/programming/CL/elephant-work/src/db-bdb/sleepycat.lisp > *** /home/viper/programming/CL/elephant/src/db-bdb/sleepycat.lisp > 2006-04-29 > 19:03:49.000000000 -0600 > --- > /home/viper/programming/CL/elephant-work/src/db-bdb/sleepycat.lisp > 2006-09-02 > 22:05:23.000000000 -0600 > *************** > *** 119,151 **** > (defconstant DB-UNKNOWN 5) > > (defconstant DB_AUTO_COMMIT #x1000000) > ! (defconstant DB_JOINENV #x0040000) > ! (defconstant DB_INIT_CDB #x0001000) > ! (defconstant DB_INIT_LOCK #x0002000) > ! (defconstant DB_INIT_LOG #x0004000) > ! (defconstant DB_INIT_MPOOL #x0008000) > ! (defconstant DB_INIT_REP #x0010000) > ! (defconstant DB_INIT_TXN #x0020000) > (defconstant DB_RECOVER #x0000020) > (defconstant DB_RECOVER_FATAL #x0200000) > (defconstant DB_LOCKDOWN #x0080000) > (defconstant DB_PRIVATE #x0100000) > ! (defconstant DB_SYSTEM_MEM #x0400000) > (defconstant DB_THREAD #x0000040) > (defconstant DB_FORCE #x0000004) > ! (defconstant DB_DEGREE_2 #x2000000) > (defconstant DB_DIRTY_READ #x4000000) > (defconstant DB_CREATE #x0000001) > ! (defconstant DB_EXCL #x0001000) > (defconstant DB_NOMMAP #x0000008) > (defconstant DB_RDONLY #x0000010) > (defconstant DB_TRUNCATE #x0000080) > (defconstant DB_TXN_NOSYNC #x0000100) > ! (defconstant DB_TXN_NOWAIT #x0001000) > ! (defconstant DB_TXN_SYNC #x0002000) > (defconstant DB_LOCK_NOWAIT #x002) > ! (defconstant DB_DUP #x0000002) > ! (defconstant DB_DUPSORT #x0000004) > > (defconstant DB_CURRENT 7) > (defconstant DB_FIRST 9) > --- 119,152 ---- > (defconstant DB-UNKNOWN 5) > > (defconstant DB_AUTO_COMMIT #x1000000) > ! (defconstant DB_JOINENV #x0000000) > ! (defconstant DB_INIT_CDB #x0002000) > ! (defconstant DB_INIT_LOCK #x0004000) > ! (defconstant DB_INIT_LOG #x0008000) > ! (defconstant DB_INIT_MPOOL #x0010000) > ! (defconstant DB_INIT_REP #x0020000) > ! (defconstant DB_INIT_TXN #x0040000) > (defconstant DB_RECOVER #x0000020) > (defconstant DB_RECOVER_FATAL #x0200000) > (defconstant DB_LOCKDOWN #x0080000) > (defconstant DB_PRIVATE #x0100000) > ! (defconstant DB_SYSTEM_MEM #x0800000) > (defconstant DB_THREAD #x0000040) > (defconstant DB_FORCE #x0000004) > ! (defconstant DB_DEGREE_2 #x2000000) ;; DEPRECATED, now called > DB_READ_COMMITTED > ! (defconstant DB_READ_COMMITTED #x2000000) > (defconstant DB_DIRTY_READ #x4000000) > (defconstant DB_CREATE #x0000001) > ! (defconstant DB_EXCL #x0002000) > (defconstant DB_NOMMAP #x0000008) > (defconstant DB_RDONLY #x0000010) > (defconstant DB_TRUNCATE #x0000080) > (defconstant DB_TXN_NOSYNC #x0000100) > ! (defconstant DB_TXN_NOWAIT #x0002000) > ! (defconstant DB_TXN_SYNC #x0004000) > (defconstant DB_LOCK_NOWAIT #x002) > ! (defconstant DB_DUP #x0004000) > ! (defconstant DB_DUPSORT #x0008000) > > (defconstant DB_CURRENT 7) > (defconstant DB_FIRST 9) > > Diff finished at Sat Sep 2 22:26:08 > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Mon Sep 4 05:02:11 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 04 Sep 2006 01:02:11 -0400 Subject: [elephant-devel] Berkeley DB 4.4 Patch in HEAD + improvements Message-ID: <44FBB353.4030407@csail.mit.edu> Recent CVS updates encompass: - I added Vladimir's BDB 4.3->4.4 patch to the repository - Fixed the bugs Vladimir reported; current state is bug free on Allegro/Mac OS/BDB - Re-organized sleepycat.lisp a bit to accommodate the BDB 4.4 convention changes - Added a new method to the backend API 'optimize-storage'. - Made serialization properly re-entrant (circularity-hash was not re-entrant) - BUGFIX: add-index with the populate option created incomplete secondary indicies due to a non-local exit within with-transaction. This was introduced by an earlier bugfix to add-index to address out of memory errors on very large indices. - Updated the TODO list for 0.6.1 BDB 4.4 upgrade notes: Existing users of the Berkeley DB backend using v4.3 MUST upgrade to Berkeley DB v4.4.20 or later. Upgrade is painless and quite safe. Instructions are included in the file "UPGRADE-BDB" in the elephant root. I will also send an e-mail update to the list shortly. New API command (exploit new feature in BDB 4.4): 'optimize-storage' is currently implemented only on BDB backends for now and takes the two flags described in Berkeley API docs. I haven't figured out a clean way to map lisp start/stop keys to database keys so do not use the stop/start keywords for now. Using this API function with the default flags and store-controller will cause all empty pages to be released back to the file system. I did not add a test case for this command as yet, but I did run a test DB to confirm that it reclaims storage. Create a bunch of large persistent objects, check out the DB file sizes, delete them using delete-instances, call optimize-storage and then check out the file sizes again. If you've been working with really large DB's, this is a sweet feature. Thanks again to Vladimir for taking the time to look into this and submit the patch! Cheers, Ian From eslick at csail.mit.edu Mon Sep 4 05:03:07 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 04 Sep 2006 01:03:07 -0400 Subject: [elephant-devel] HEAD upgraded to BDB 4.4 / Requires Upgrade Procedure Message-ID: <44FBB38B.6010408@csail.mit.edu> If you are working from the latest HEAD and using BDB you'll need to take the following steps to upgrade your BDB databases to work with check-ins as of September 3rd) [This info can be found in UPGRADE-BDB in the elephant root directory] 1) Install BDB 4.4.20 or later just as you installed BDB 4.3.x 2) Pull the latest HEAD from CVS 3) Update config.lisp and Makefile in elephant root to point to the appropriate directories 4) Rebuild elephant C libraries In the root directory: > make clean > make > make bdb 5) Upgrade your database directory (only log files need updating) >From Sleepycat documentation: 1. Shut down the old version of the application. 2. Run recovery on the database environment using the DB_ENV->open method or the db_recover utility. 3. Remove any Berkeley DB environment using the DB_ENV->remove method or an appropriate system utility. 4. Archive the database environment for catastrophic recovery. See Archival procedures for more information. 5. Recompile and install the new version of the application. 6. Force a checkpoint using the DB_ENV->txn_checkpoint method or the db_checkpoint utility. 7. Restart the application. A known good procedure: 1. Cleanly exit lisp/elephant application 2. Run 'db_recover' in database directory using 4.3 tools 3. This will remove the environment 3.b (optional) Run 'db_checkpoint -1' and 'db_archive -d' to checkpoint and update db files to latest log. This will snapshot the DB and allow you to backup less data, but it makes catastrophic recovery to any time before the snapshot impossible as you are deleting history with the '-d' option so exclude that if you are conservative or aren't sure what you're doing. 4. Copy your database files and all log files to a backup 5. Run 'db_checkpoint -1' using 4.4 tools (ignore error message) 6. Restart lisp, reload application and ensure that the latest elephant source has been fully recompiled 6) Connect to your DB, all should be well! This procedure worked for my Mac OS X upgrade from BDB 4.3 using the latest HEAD on a very large, complex DB. From eslick at csail.mit.edu Mon Sep 4 05:10:46 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 04 Sep 2006 01:10:46 -0400 Subject: [elephant-devel] Berkeley DB 4.4.20 Patch In-Reply-To: <44FBA252.5030209@csail.mit.edu> References: <44FBA252.5030209@csail.mit.edu> Message-ID: <44FBB556.8090507@csail.mit.edu> Ah, this was a false alarm. Vladimir, the latest HEAD should fix the problems you reported. As mentioned in my last note this was actually a bug I introduced in fixing some memory problems that crop up when using add-index on a large primary index with the populate option (it used to exhaust BDB's transaction memory so I had to break up the transaction into chunks). I accidentally introduced a non-local exit through with-transaction leading to the last transaction to populate the secondary index aborting instead of committing. This only affects the BDB backend. For the record, the current measure of a successful body within (with-transaction ()) is an ordinary exit. _ANY_ non-local exit such as throw, error, etc as well as goto labels, or (return-from ) all result in aborts. More technically the test is written as: (with-transaction () ) => (unwind-protect (prog1 (setf success t) (db-transaction-commit ...)) (unless success (db-transaction-abort ...))) Should this policy be changed? Should all conditions and throws result in aborts but other non-local exits in success? Regards, Ian Ian Eslick wrote: > Vladimir, > > I applied your changes, cleaned up some other changes for a check in. I > found the same errors as you on Mac OS using Allegro. Digging deeper > into this it appears there are genuine issues cropping up in these two > tests. > > It appears that a pcursor-next operation returns nil instead of t and > prematurely terminates the walk of an index. This may be related to the > advertised change in the auto-commit behavior. Auto-commit used to be a > default that could be overridden within a transaction. The new behavior > is global so either you always auto-commit or you have to wrap > transactions around everything. BDB users relying on the old > interleaving behavior may be heavily effected by the 4.4 upgrade. > > I'll continue to look into this tomorrow so I better understand the > implications. > > Regards, > Ian > > Vladimir Sedach wrote: > >> Hi everybody, >> >> Unless the BDB people are lying in their Changelog, the only thing >> that changed between 4.3 and 4.4 as far as Elephant should be >> concerned are the constants in db.h. Diff information provided below. >> I don't use Elephant for anything right now (but I am planning to in >> the future, and what better way to get familiar than fixing things >> like this?), so the test suite is the only thing I have to go by. For >> both BDB 4.3 and 4.4, do-backend-tests passes everything but PCURSOR2 >> and INDEXING-REDEF-CLASS. Should I be worried? >> >> Vladimir >> >> cd /home/viper/programming/CL/elephant-work/src/db-bdb/ >> diff -c /home/viper/programming/CL/elephant/src/db-bdb/sleepycat.lisp >> /home/viper/programming/CL/elephant-work/src/db-bdb/sleepycat.lisp >> *** /home/viper/programming/CL/elephant/src/db-bdb/sleepycat.lisp >> 2006-04-29 >> 19:03:49.000000000 -0600 >> --- >> /home/viper/programming/CL/elephant-work/src/db-bdb/sleepycat.lisp >> 2006-09-02 >> 22:05:23.000000000 -0600 >> *************** >> *** 119,151 **** >> (defconstant DB-UNKNOWN 5) >> >> (defconstant DB_AUTO_COMMIT #x1000000) >> ! (defconstant DB_JOINENV #x0040000) >> ! (defconstant DB_INIT_CDB #x0001000) >> ! (defconstant DB_INIT_LOCK #x0002000) >> ! (defconstant DB_INIT_LOG #x0004000) >> ! (defconstant DB_INIT_MPOOL #x0008000) >> ! (defconstant DB_INIT_REP #x0010000) >> ! (defconstant DB_INIT_TXN #x0020000) >> (defconstant DB_RECOVER #x0000020) >> (defconstant DB_RECOVER_FATAL #x0200000) >> (defconstant DB_LOCKDOWN #x0080000) >> (defconstant DB_PRIVATE #x0100000) >> ! (defconstant DB_SYSTEM_MEM #x0400000) >> (defconstant DB_THREAD #x0000040) >> (defconstant DB_FORCE #x0000004) >> ! (defconstant DB_DEGREE_2 #x2000000) >> (defconstant DB_DIRTY_READ #x4000000) >> (defconstant DB_CREATE #x0000001) >> ! (defconstant DB_EXCL #x0001000) >> (defconstant DB_NOMMAP #x0000008) >> (defconstant DB_RDONLY #x0000010) >> (defconstant DB_TRUNCATE #x0000080) >> (defconstant DB_TXN_NOSYNC #x0000100) >> ! (defconstant DB_TXN_NOWAIT #x0001000) >> ! (defconstant DB_TXN_SYNC #x0002000) >> (defconstant DB_LOCK_NOWAIT #x002) >> ! (defconstant DB_DUP #x0000002) >> ! (defconstant DB_DUPSORT #x0000004) >> >> (defconstant DB_CURRENT 7) >> (defconstant DB_FIRST 9) >> --- 119,152 ---- >> (defconstant DB-UNKNOWN 5) >> >> (defconstant DB_AUTO_COMMIT #x1000000) >> ! (defconstant DB_JOINENV #x0000000) >> ! (defconstant DB_INIT_CDB #x0002000) >> ! (defconstant DB_INIT_LOCK #x0004000) >> ! (defconstant DB_INIT_LOG #x0008000) >> ! (defconstant DB_INIT_MPOOL #x0010000) >> ! (defconstant DB_INIT_REP #x0020000) >> ! (defconstant DB_INIT_TXN #x0040000) >> (defconstant DB_RECOVER #x0000020) >> (defconstant DB_RECOVER_FATAL #x0200000) >> (defconstant DB_LOCKDOWN #x0080000) >> (defconstant DB_PRIVATE #x0100000) >> ! (defconstant DB_SYSTEM_MEM #x0800000) >> (defconstant DB_THREAD #x0000040) >> (defconstant DB_FORCE #x0000004) >> ! (defconstant DB_DEGREE_2 #x2000000) ;; DEPRECATED, now called >> DB_READ_COMMITTED >> ! (defconstant DB_READ_COMMITTED #x2000000) >> (defconstant DB_DIRTY_READ #x4000000) >> (defconstant DB_CREATE #x0000001) >> ! (defconstant DB_EXCL #x0002000) >> (defconstant DB_NOMMAP #x0000008) >> (defconstant DB_RDONLY #x0000010) >> (defconstant DB_TRUNCATE #x0000080) >> (defconstant DB_TXN_NOSYNC #x0000100) >> ! (defconstant DB_TXN_NOWAIT #x0002000) >> ! (defconstant DB_TXN_SYNC #x0004000) >> (defconstant DB_LOCK_NOWAIT #x002) >> ! (defconstant DB_DUP #x0004000) >> ! (defconstant DB_DUPSORT #x0008000) >> >> (defconstant DB_CURRENT 7) >> (defconstant DB_FIRST 9) >> >> Diff finished at Sat Sep 2 22:26:08 >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From vsedach at gmail.com Tue Sep 5 02:30:33 2006 From: vsedach at gmail.com (Vladimir Sedach) Date: Mon, 4 Sep 2006 20:30:33 -0600 Subject: [elephant-devel] Berkeley DB 4.4.20 Patch In-Reply-To: <44FBB556.8090507@csail.mit.edu> References: <44FBA252.5030209@csail.mit.edu> <44FBB556.8090507@csail.mit.edu> Message-ID: > For the record, the current measure of a successful body within > (with-transaction ()) is an ordinary exit. _ANY_ non-local exit such as > throw, error, etc as well as goto labels, or (return-from ) > all result in aborts. More technically the test is written as: > > (with-transaction () > ) > > => > > (unwind-protect > (prog1 > > (setf success t) > (db-transaction-commit ...)) > (unless success > (db-transaction-abort ...))) > > Should this policy be changed? Should all conditions and throws result > in aborts but other non-local exits in success? First of all, unless I am missing something, shouldn't it be (progn (db-transaction-commit ...) [and then] (setf success t))? I think that the policy of aborting the transaction on all non-local exits as it is now is a good idea. Maybe will funcall some lambda passed to it that does weird things with the control flow which could lead to hard-to-find bugs. I also have a feeling this may lead to such bugs with code that tries to fake continuations in some instances, but I can't think of any examples. However, then there should be a protocol to let programmers deal with transactions in hairy code themselves (hey, transactions are a part of the business logic or whatever... :)). Here is what I propose: (defun close-transaction () ;; this gets exported (db-transaction-commit :transaction *txn* :txn-nosync *txn-nosync* :txn-sync *txn-sync*) (setq *success* t)) now (with-transaction ) becomes: (let ((*success* nil) (*txn ... ) (declare (special *success*) (special *txn... ) (unwind-protect (progn (unless *success* (close-transaction))) ;; this makes sure we don't commit twice (unless *success* (abort-transaction... )) So when you want to do something to leave control from without reaching the end, you can call (elephant:close-transaction) or whatever (maybe backend-close-transaction is a better name?) to explicitly close that transaction. This also handles the problem that somebody may want to raise an error or something but still want the transaction to go through. I would implement the above myself, but I can't get Elephant from the CVS circa right now to work. First thing is that in serializer.lisp, in functions get-circularity-hash and release-circularity-hash, there is the (#+allegro mp::with-process-lock (*circularity-lock*) (pop/push...)), which SBCL doesn't like, so you have to do #+allegro (mp::with-process lock... (pop/push...)) #-allegro (pop/push...). Then in sleepycat.lisp, (def-function ("db_compact" ... should have :returning :pointer-void (or maybe it is better for def-function lambda-list to default &key returning to &key (returning :pointer-void)?). After that everything seems to compile but nothing seems to work. I'm not quite sure why, and I'm too tired to find out right now. Vladimir From eslick at csail.mit.edu Tue Sep 5 02:37:06 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 04 Sep 2006 22:37:06 -0400 Subject: [elephant-devel] Berkeley DB 4.4.20 Patch In-Reply-To: References: <44FBA252.5030209@csail.mit.edu> <44FBB556.8090507@csail.mit.edu> Message-ID: <44FCE2D2.8080404@csail.mit.edu> Oops - I forgot about that hack. In order to make serialization thread safe I need to use locks which are non-standard. I'll hack something for SBCL in minute or two. If you want to manually controll transaction you can use the following protocol: (start-ele-transaction) (transaction-commit) (transaction-abort) inside of your own logic. with-transaction is a convenience macro with a specific control policy. We can leave that policy and let more complex programs make it all explicit (and suffer the consequences). Perhaps a 'condition on abort' on a non-local exit (that isn't a thrown condition) from with-transaction would help catch any annoying bugs caused by silent aborts. Ian Vladimir Sedach wrote: >> For the record, the current measure of a successful body within >> (with-transaction ()) is an ordinary exit. _ANY_ non-local exit such as >> throw, error, etc as well as goto labels, or (return-from ) >> all result in aborts. More technically the test is written as: >> >> (with-transaction () >> ) >> >> => >> >> (unwind-protect >> (prog1 >> >> (setf success t) >> (db-transaction-commit ...)) >> (unless success >> (db-transaction-abort ...))) >> >> Should this policy be changed? Should all conditions and throws result >> in aborts but other non-local exits in success? > > First of all, unless I am missing something, shouldn't it be (progn > (db-transaction-commit ...) [and then] (setf success t))? > > I think that the policy of aborting the transaction on all non-local > exits as it is now is a good idea. Maybe will funcall some > lambda passed to it that does weird things with the control flow which > could lead to hard-to-find bugs. I also have a feeling this may lead > to such bugs with code that tries to fake continuations in some > instances, but I can't think of any examples. However, then there > should be a protocol to let programmers deal with transactions in > hairy code themselves (hey, transactions are a part of the business > logic or whatever... :)). Here is what I propose: > > (defun close-transaction () ;; this gets exported > (db-transaction-commit :transaction *txn* > :txn-nosync *txn-nosync* > :txn-sync *txn-sync*) > (setq *success* t)) > > now (with-transaction ) becomes: > > (let ((*success* nil) (*txn ... ) > (declare (special *success*) (special *txn... ) > (unwind-protect > (progn > > (unless *success* (close-transaction))) ;; this makes sure we > don't commit twice > (unless *success* (abort-transaction... )) > > So when you want to do something to leave control from without > reaching the end, you can call (elephant:close-transaction) or > whatever (maybe backend-close-transaction is a better name?) to > explicitly close that transaction. This also handles the problem that > somebody may want to raise an error or something but still want the > transaction to go through. > > I would implement the above myself, but I can't get Elephant from the > CVS circa right now to work. First thing is that in serializer.lisp, > in functions get-circularity-hash and release-circularity-hash, there > is the (#+allegro mp::with-process-lock (*circularity-lock*) > (pop/push...)), which SBCL doesn't like, so you have to do #+allegro > (mp::with-process lock... (pop/push...)) #-allegro (pop/push...). Then > in sleepycat.lisp, (def-function ("db_compact" ... should have > :returning :pointer-void (or maybe it is better for def-function > lambda-list to default &key returning to &key (returning > :pointer-void)?). After that everything seems to compile but nothing > seems to work. I'm not quite sure why, and I'm too tired to find out > right now. > > Vladimir > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Tue Sep 5 03:21:42 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 04 Sep 2006 23:21:42 -0400 Subject: [elephant-devel] Berkeley DB 4.4.20 Patch In-Reply-To: <44FCE2D2.8080404@csail.mit.edu> References: <44FBA252.5030209@csail.mit.edu> <44FBB556.8090507@csail.mit.edu> <44FCE2D2.8080404@csail.mit.edu> Message-ID: <44FCED46.9010601@csail.mit.edu> Ok, I just checked in a fix that should work for SBCL, CMU, LispWorks and MCL and that I've done a preliminary test of under SBCL/Mac OS I also added a simple adaptation algorithm so that the system slowly moves towards the average object collection size. It will naturally tune to the thread usage patterns. You will probably never need this unless you store lists, arrays of type t or hash tables full of dozens or hundreds of normal objects. Large complex lists will also cause the hash tables to grow as an entry is required for every cons cell. Parameters can be tuned in variables.lisp. Regards, Ian Ian Eslick wrote: > Oops - I forgot about that hack. In order to make serialization thread > safe I need to use locks which are non-standard. I'll hack something > for SBCL in minute or two. > > If you want to manually controll transaction you can use the following > protocol: > > (start-ele-transaction) > (transaction-commit) > That's actually (abort-transaction) > (transaction-abort) > > and (commit-transaction) > inside of your own logic. > > with-transaction is a convenience macro with a specific control policy. > We can leave that policy and let more > complex programs make it all explicit (and suffer the consequences). > > Perhaps a 'condition on abort' on a non-local exit (that isn't a thrown > condition) from with-transaction would help catch any annoying bugs > caused by silent aborts. > > Ian > > Vladimir Sedach wrote: > >>> For the record, the current measure of a successful body within >>> (with-transaction ()) is an ordinary exit. _ANY_ non-local exit such as >>> throw, error, etc as well as goto labels, or (return-from ) >>> all result in aborts. More technically the test is written as: >>> >>> (with-transaction () >>> ) >>> >>> => >>> >>> (unwind-protect >>> (prog1 >>> >>> (setf success t) >>> (db-transaction-commit ...)) >>> (unless success >>> (db-transaction-abort ...))) >>> >>> Should this policy be changed? Should all conditions and throws result >>> in aborts but other non-local exits in success? >>> >> First of all, unless I am missing something, shouldn't it be (progn >> (db-transaction-commit ...) [and then] (setf success t))? >> >> I think that the policy of aborting the transaction on all non-local >> exits as it is now is a good idea. Maybe will funcall some >> lambda passed to it that does weird things with the control flow which >> could lead to hard-to-find bugs. I also have a feeling this may lead >> to such bugs with code that tries to fake continuations in some >> instances, but I can't think of any examples. However, then there >> should be a protocol to let programmers deal with transactions in >> hairy code themselves (hey, transactions are a part of the business >> logic or whatever... :)). Here is what I propose: >> >> (defun close-transaction () ;; this gets exported >> (db-transaction-commit :transaction *txn* >> :txn-nosync *txn-nosync* >> :txn-sync *txn-sync*) >> (setq *success* t)) >> >> now (with-transaction ) becomes: >> >> (let ((*success* nil) (*txn ... ) >> (declare (special *success*) (special *txn... ) >> (unwind-protect >> (progn >> >> (unless *success* (close-transaction))) ;; this makes sure we >> don't commit twice >> (unless *success* (abort-transaction... )) >> >> So when you want to do something to leave control from without >> reaching the end, you can call (elephant:close-transaction) or >> whatever (maybe backend-close-transaction is a better name?) to >> explicitly close that transaction. This also handles the problem that >> somebody may want to raise an error or something but still want the >> transaction to go through. >> >> I would implement the above myself, but I can't get Elephant from the >> CVS circa right now to work. First thing is that in serializer.lisp, >> in functions get-circularity-hash and release-circularity-hash, there >> is the (#+allegro mp::with-process-lock (*circularity-lock*) >> (pop/push...)), which SBCL doesn't like, so you have to do #+allegro >> (mp::with-process lock... (pop/push...)) #-allegro (pop/push...). Then >> in sleepycat.lisp, (def-function ("db_compact" ... should have >> :returning :pointer-void (or maybe it is better for def-function >> lambda-list to default &key returning to &key (returning >> :pointer-void)?). After that everything seems to compile but nothing >> seems to work. I'm not quite sure why, and I'm too tired to find out >> right now. >> >> Vladimir >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > From eslick at csail.mit.edu Sat Sep 9 16:57:49 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 09 Sep 2006 12:57:49 -0400 Subject: [elephant-devel] A Modest Proposal Message-ID: <4502F28D.2050405@csail.mit.edu> Dear Developers, Every time I have to work directly with BTrees I find myself become quite vexed. When I was playing around with factoring out the backends I had a first hand opportunity to observe how Robert had to play games to get the SQL back end to implement the BTree interface. It strikes me that "the right thing" is to define a more abstract interface that is exported from the elephant package by default and plan to deprecate the BTree interface. I would like to propose to put a new table/index design into the roadmap, along with a period of guaranteed backwards compatibility for the BTree interface to expire in a year or so and which does not bind any new backends to support it. Existing systems which directly use the BTree interface can still access it but perhaps by importing it directly from the backend or via a special package. So what is the right persistent collection model? We want something that plays nicely with transactions, doesn't place any onerous performance burden on backends and plays nicely with the existing BDB and SQL implementations. - An iterator model? Problematic because iterators are meant to be something akin to closures, an object that encapsulates the state of an iteration. It would be easy to move an iterator around outside the dynamic extent of a transaction which, from experience, would lead to crashes in the BDB backend which is deadly for new users and annoying to me. - A more abstract table class? What methods? - (defclass table ...) - (get-value key table) => value - (find-key value table) => key, pointer - (do-table () ) - (defclass table-pointer ...) - (get pointer) - (set pointer) - (predecessor pointer &key (skip-dups nil) (keys-only nil)) => key, value ; update pointer - (successor pointer &key (skip-dups nil) (keys-only nil)) => key, value ; update pointer This proposes to add a layer of in-memory bookkeeping that can do the right thing if the bookkeeping objects fall outside of the scope of a given transaction. (For example, cursors are released when the transaction terminates regardless of whether with-btree-cursor was used and can be re-opened if the pointers are used in a new transaction - traversal state can now persist cheaply across transactions). In fact this would allow a simple iterator model to be supported as well as the kind of multiple-pointer traversals common in compiled queries. This barrier between the user and cursors would simplify use of the BDB backend, at a minimum. - Do we want to implement something like the AllegroStore persistent set object? This would be something that only keeps OID's so it's lightweight to serialize, store and manipulate but the retrieval function de-references appropriately. I need to do some more research into all this, but wanted to put my thoughts out there and invite comment. Ian From read at robertlread.net Sat Sep 9 17:15:25 2006 From: read at robertlread.net (Robert L. Read) Date: Sat, 09 Sep 2006 12:15:25 -0500 Subject: [elephant-devel] A Modest Proposal In-Reply-To: <4502F28D.2050405@csail.mit.edu> References: <4502F28D.2050405@csail.mit.edu> Message-ID: <1157822125.8466.265.camel@localhost.localdomain> On Sat, 2006-09-09 at 12:57 -0400, Ian Eslick wrote: > So what is the right persistent collection model? We want something > that plays nicely with transactions, doesn't place any onerous > performance burden on backends and plays nicely with the existing BDB > and SQL implementations. > I strongly support a more abstract model, and I would recommend the "Dictionary" as the correct abstract model. (That's what we called it back in CS when I was in college.) The hashtable is an implementation of the Dictionary that is so close it's hard to remember that Dictionary is abstract and hashtable is just one of several implementation mechanisms ( for example, a btree is another.) I would tend to oppose any notion of "tables". To me, that is just circling back around to a relational model; CLSQL and its object-relational mapping partially covers that territory. The "btree" is not very abstract --- it is what BDB uses, and therefore was a fine object to expose at the time. Dictionaries of course can support iterators, and one has to have a way iterating over all of the objects in the collection. I tend to prefer the "apply this function to every object in a certain model" to the language of iterators, but in practice it's mostly the same. I only briefly looked at the AllegroStore stuff; but the notion of a "persistent set" seems quite clean. But please don't let my comments dissuade others from commenting. -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Sat Sep 9 17:21:05 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 09 Sep 2006 13:21:05 -0400 Subject: [elephant-devel] A Modest Proposal In-Reply-To: <1157822125.8466.265.camel@localhost.localdomain> References: <4502F28D.2050405@csail.mit.edu> <1157822125.8466.265.camel@localhost.localdomain> Message-ID: <4502F801.9050505@csail.mit.edu> I think I meant 'dictionary' when I said table - I don't use relational databases very often so the association isn't that strong for me! However BTrees support two extra functions that are important for writing query languages and probably in other operations. One is that the traversal is _ordered_, a second is an ability to index _to a point in an ordering_ and perform successor/predecessor operations from that point. (i.e. retrieve objects with slot A between 10 and 20 and a pointer in slot B to Obj C). Given that you're seeking to disk, these optimizations are critical to performance on large datasets. Persistent sets are clean, but completely unordered. I think the sequence datastructure in BDB could be used to implement something like this, but for now it could be done easily, but more expensively, on top of an index. The nice thing, I imagine, is that they just contain ID's so you can do intersection/difference/append quite easily. I think you could do the same with SQL, although I don't know if there's an efficient SQL implementation unless you can incrementally update the internal structure of blobs without reading all the data into memory. Ian Robert L. Read wrote: > On Sat, 2006-09-09 at 12:57 -0400, Ian Eslick wrote: >> So what is the right persistent collection model? We want something >> that plays nicely with transactions, doesn't place any onerous >> performance burden on backends and plays nicely with the existing BDB >> and SQL implementations. >> > > I strongly support a more abstract model, and I would recommend the > "Dictionary" as the > correct abstract model. (That's what we called it back in CS when I > was in college.) The > hashtable is an implementation of the Dictionary that is so close it's > hard to remember that > Dictionary is abstract and hashtable is just one of several > implementation mechanisms ( > for example, a btree is another.) > > I would tend to oppose any notion of "tables". To me, that is just > circling back around to > a relational model; CLSQL and its object-relational mapping partially > covers that territory. > > The "btree" is not very abstract --- it is what BDB uses, and > therefore was a fine object > to expose at the time. > > Dictionaries of course can support iterators, and one has to have a > way iterating over > all of the objects in the collection. I tend to prefer the "apply > this function to every object > in a certain model" to the language of iterators, but in practice it's > mostly the same. > > I only briefly looked at the AllegroStore stuff; but the notion of a > "persistent set" seems > quite clean. > > But please don't let my comments dissuade others from commenting. > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at csail.mit.edu Sat Sep 9 18:08:31 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Sat, 09 Sep 2006 14:08:31 -0400 Subject: [elephant-devel] Persistance and Model Validations In-Reply-To: <1155332422.4658.1490.camel@localhost.localdomain> References: <1C6C3914-0E35-4F08-886F-E60EC0CBD3B2@infoway.net> <1155332422.4658.1490.camel@localhost.localdomain> Message-ID: <4503031F.7040803@csail.mit.edu> Took me awhile to get back to this: There are no hooks today other than in the backend, that you could wrap around the backend start, abort and commit transaction calls but there would be no way to know what is committed because the transactions are at the data level, not the object level. This kind of functionality would probably be easier to add to Rucksack, but as Robert says you could write your own protocol and macro that leveraged the transaction machinery. Pardon the naming convention: ;; User IF (defparameter *active-objects* nil) (def-validator (type) ) (def-fetch (id) ) (with-validation () ) ;; Implementing with-validation (defmacro with-validation (() &body body) `(let ((*active-objects* nil)) (declare (special *active-objects*)) (unwind-protect (progn , at body) (let ((abort-p nil)) (dolist (obj *active-objects*) (unless (validate obj) (setq abort-p t))) (dolist (obj *active-objects*) (if abort-p (delete obj) (save obj))) (if abort-p (abort-transaction) (commit-transaction))))) If you've defined a validator for an object, you can use with-validators to build a transactional function that retrieves one or more objects, does some computation and then runs validation on all fetched objects prior to committing them. This requires that you have a way of keeping track of what was touched (via registration on using 'fetch') so it would have to build into your application API that required user use that function so you can guarantee to call validate, save or delete on all modified objects. Following this model, a user can implement :before and :after methods around validate, delete and save specialized on the object classes in question and then take appropriate action. For example if you wanted to index an object when it was saved as in Robert's example you would (defmethod save :after ()). Because of the fine-grained, storage-level model of data in Elephant, delete and save do not actually do the work to revert or store the data, but can be used as a change point for user overloading. Again, Rucksack has the same object-centric view of the world that I think the Ruby system has so would be easier to adapt to this worldview. You can also use :around if you need to get at the results of calling validate. (FYI - the above code fragments don't handle other cases requiring aborts (non-local flow of control, etc) nor does it account for what happens if a user calls (abort-transaction) manually.) Ian Robert L. Read wrote: > Just a quick comment on this --- Ian may have a better idea. > > Yes, if we offered a nice clean place to put all of those, it would > make it obvious > where to put validation. (I will certainly think about that in the > future.) > > Here is a technique that I use, that is very effective for me, but not > generally > applicable: I use DCM, which is in the contrib directory and > basically provides > wrappers for the create/read/update/destory operations, similar to > those you > name below. I use :after methods on these operations (specialized on > Director > classes, since it is generally class dependedent) to send objects to a > text indexer > in a background thread whenever they change. (I was using Lucene-ws > but now > use montezuma; it makes sense to do these operations in a back-ground > thread since > a web-service call might block the thread that needs to get a response > on to the > glass as fast as possible.) > > This is an extraordinarily powerful and elegant feature of LISP. If > you have a method > that gives you a "change point" (I believe that is what you are asking > for) then you can > easily put a :before method in a completely different file that > performs validation and > signals an exception or takes some other action that interrupts the > intended operation. > One of the beauties of this is that the validation rules can be easily > kept quite separate from the rest of the code. > > I suppose this same technique could be applied even if you do not have > clearly defined > methods; but personally I consider this clarity to be one of the > advantages of the DCM code. > Of course, one can more or less easily create your own validation > points, without becoming > dependeing on the DCM code. > > Here's my example code: > (defmethod register-obj :after ((dir indexing-director) (mo > managed-object)) > (pub-and-proc (list "indexdocument" dir (k (mid mo)))) > ) > > (defmethod delete-obj :before ((dir indexing-director) (id key)) > (pub-and-proc (list "removedocument" dir (k id))) > ) > "register-obj" and "delete-obj" are defined in DCM. > > > On Fri, 2006-08-11 at 17:17 -0400, Daniel Salama wrote: >> Hi, >> >> This may be more of a suggestion since I think I already know the >> answer. >> >> I come from the Ruby On Rails world and, as some of you may know, >> their Active Record module provides hooks that allow the automatic >> and customized evaluation of validation rules at different stages of >> the committing process. >> >> At first, I was going to as how can I hook validation rules into the >> persistent machinery. I guess I could (should) always wrap my commits >> in transactions and then I could perform the validation process >> within the transaction and, therefore, I could commit or abort >> depending on the validation results. >> >> However, as my Lisp knowledge is still limited, I'd like to suggest >> for some future version of Elephant, the inclusion of some type of >> macro that works somehow like a with-transaction type operation. This >> macro could support generic methods that could be "customized" per >> class just like CLOS generic methods to support model validation. >> Just for reference, Rails validation hooks support, among others: >> >> - before validation >> - after validation >> - before create >> - after create >> - before save >> - after save >> - before destroy >> - after destroy >> >> Maybe some of this functionality could be implemented in the near >> future. Maybe it's not that big of a deal, but, since my Lisp >> knowledge is limited, I am just throwing this suggestion. >> >> Then again, maybe there is a better way to do it and I just have too >> much of a Rails mentality for the time being :) >> >> Thanks, >> Daniel >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From pinterface at gmail.com Tue Sep 12 08:47:01 2006 From: pinterface at gmail.com (pinterface) Date: Tue, 12 Sep 2006 08:47:01 -0000 Subject: [elephant-devel] config.lisp patch; backend-tests question Message-ID: <2b4301c6d648$04a88460$3100a8c0@trinity> Howdy all! I'm new to elephant (just installed and compiled it over the weekend), and a couple parts of the installation process struck me as being particularly unlispy: * a Makefile (and needing to modify it) * editing config.lisp Attached is a fairly simple patch to allow something like ;; set to the location of the ubuntu BDB-4.4 package (setf elephant:*sleepycat-foreign-library-path* "/usr/lib/libdb-4.4.so") (asdf:operate 'asdf:load-op :ele-bdb) to work, rather than requiring direct modification of config.lisp, which is prone to having issues with cvs update. I'd like to eliminate the makefile entirely and have ASDF handle the compilation of C files itself, rather than running through make; that way, .so files will end up in the correct locations under systems such as asdf-binary-locations (osicat does this, and it's quite handy not having to shuffle .so files around myself, which I tend to forget). That'll be a bit more involved and require delving into ASDF a good deal farther than I've been before, so we'll see how long it takes me to figure out. (Assuming such a patch would even be welcome, of course.) Anyway... After some minor confusion caused by backend tests only succeeding once[1], I appear to have a usable installation and am looking forward to playing with it more in depth. [1] Are the BDB backend tests expected to fail the second time when run multiple times? i.e., (progn (do-backend-tests) ; pass (do-backend-tests)) ; fail I figure by the existence of delscript.sh it's normal behavior (or at least not worryingly abnormal), but not seeing it mentioned in any docs thought I'd check, just to be sure. That about covers it for now, I think. -pix Relevant system specs, for reference: * SBCL (from CVS) * Elephant (from CVS) * UFFI (latest version) * asdf-binary-locations (from darcs repo) * BerkeleyDB-4.4 (Ubuntu universe package) -------------- next part -------------- A non-text attachment was scrubbed... Name: elephant.patch Type: application/octet-stream Size: 2115 bytes Desc: not available URL: From read at robertlread.net Wed Sep 13 14:52:55 2006 From: read at robertlread.net (Robert L. Read) Date: Wed, 13 Sep 2006 09:52:55 -0500 Subject: [elephant-devel] config.lisp patch; backend-tests question In-Reply-To: <2b4301c6d648$04a88460$3100a8c0@trinity> References: <2b4301c6d648$04a88460$3100a8c0@trinity> Message-ID: <1158159176.8466.422.camel@localhost.localdomain> Thank you! I'll talk to Ian about dealing with this patch; it certainly is nice to improve the installation process. Makefile is hardly offensive, but your point about smoothing the library placement is well taken. Yes, I seem to recall (it's been a while since I did this specifically for BDB) that the tests are not completely idempotent (runnable one after another.) The fact that they fail should make you doubt the quality of our testing procedure, but not the core functionality of Elephant itself; if your test are green once, you have a good install, and any remaining bugs must be considered bug in the product itself. On Tue, 2006-09-12 at 08:47 +0000, pinterface wrote: > Howdy all! > > I'm new to elephant (just installed and compiled it over the weekend), and a > couple parts of the installation process struck me as being particularly > unlispy: > * a Makefile (and needing to modify it) > * editing config.lisp > > Attached is a fairly simple patch to allow something like > ;; set to the location of the ubuntu BDB-4.4 package > (setf elephant:*sleepycat-foreign-library-path* "/usr/lib/libdb-4.4.so") > (asdf:operate 'asdf:load-op :ele-bdb) > to work, rather than requiring direct modification of config.lisp, which is > prone to having issues with cvs update. > > I'd like to eliminate the makefile entirely and have ASDF handle the > compilation of C files itself, rather than running through make; that way, > .so files will end up in the correct locations under systems such as > asdf-binary-locations (osicat does this, and it's quite handy not having to > shuffle .so files around myself, which I tend to forget). That'll be a bit > more involved and require delving into ASDF a good deal farther than I've > been before, so we'll see how long it takes me to figure out. (Assuming such > a patch would even be welcome, of course.) Anyway... > > After some minor confusion caused by backend tests only succeeding once[1], > I appear to have a usable installation and am looking forward to playing > with it more in depth. > > [1] Are the BDB backend tests expected to fail the second time when run > multiple times? > i.e., (progn (do-backend-tests) ; pass > (do-backend-tests)) ; fail > I figure by the existence of delscript.sh it's normal behavior (or at least > not worryingly abnormal), but not seeing it mentioned in any docs thought > I'd check, just to be sure. > > That about covers it for now, I think. > > -pix > > Relevant system specs, for reference: > * SBCL (from CVS) > * Elephant (from CVS) > * UFFI (latest version) > * asdf-binary-locations (from darcs repo) > * BerkeleyDB-4.4 (Ubuntu universe package) > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel -------------- next part -------------- An HTML attachment was scrubbed... URL: From eslick at csail.mit.edu Wed Sep 13 21:54:47 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Wed, 13 Sep 2006 17:54:47 -0400 Subject: [elephant-devel] config.lisp patch; backend-tests question In-Reply-To: <1158159176.8466.422.camel@localhost.localdomain> References: <2b4301c6d648$04a88460$3100a8c0@trinity> <1158159176.8466.422.camel@localhost.localdomain> Message-ID: <45087E27.30203@csail.mit.edu> I'll look at this over the weekend. I was slightly unhappy with the build process, but never enough to do anything about it so thanks for taking the initiative! There is some issues with the tests assuming that objects under test don't exist - after the test they exist so that assumption is broken. So as Robert says the tests are not idempotent. I thought I had caught all those issues for 0.6.0, but I can't guarantee that it's not happening again in the current CVS HEAD. Cheers, Ian Robert L. Read wrote: > Thank you! > > I'll talk to Ian about dealing with this patch; it certainly is nice > to improve the installation process. > Makefile is hardly offensive, but your point about smoothing the > library placement is well taken. > > Yes, I seem to recall (it's been a while since I did this specifically > for BDB) that the tests are > not completely idempotent (runnable one after another.) The fact that > they fail should make you > doubt the quality of our testing procedure, but not the core > functionality of Elephant itself; if your > test are green once, you have a good install, and any remaining bugs > must be considered bug > in the product itself. > > On Tue, 2006-09-12 at 08:47 +0000, pinterface wrote: >> Howdy all! >> >> I'm new to elephant (just installed and compiled it over the weekend), and a >> couple parts of the installation process struck me as being particularly >> unlispy: >> * a Makefile (and needing to modify it) >> * editing config.lisp >> >> Attached is a fairly simple patch to allow something like >> ;; set to the location of the ubuntu BDB-4.4 package >> (setf elephant:*sleepycat-foreign-library-path* "/usr/lib/libdb-4.4.so") >> (asdf:operate 'asdf:load-op :ele-bdb) >> to work, rather than requiring direct modification of config.lisp, which is >> prone to having issues with cvs update. >> >> I'd like to eliminate the makefile entirely and have ASDF handle the >> compilation of C files itself, rather than running through make; that way, >> .so files will end up in the correct locations under systems such as >> asdf-binary-locations (osicat does this, and it's quite handy not having to >> shuffle .so files around myself, which I tend to forget). That'll be a bit >> more involved and require delving into ASDF a good deal farther than I've >> been before, so we'll see how long it takes me to figure out. (Assuming such >> a patch would even be welcome, of course.) Anyway... >> >> After some minor confusion caused by backend tests only succeeding once[1], >> I appear to have a usable installation and am looking forward to playing >> with it more in depth. >> >> [1] Are the BDB backend tests expected to fail the second time when run >> multiple times? >> i.e., (progn (do-backend-tests) ; pass >> (do-backend-tests)) ; fail >> I figure by the existence of delscript.sh it's normal behavior (or at least >> not worryingly abnormal), but not seeing it mentioned in any docs thought >> I'd check, just to be sure. >> >> That about covers it for now, I think. >> >> -pix >> >> Relevant system specs, for reference: >> * SBCL (from CVS) >> * Elephant (from CVS) >> * UFFI (latest version) >> * asdf-binary-locations (from darcs repo) >> * BerkeleyDB-4.4 (Ubuntu universe package) >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From pinterface at gmail.com Sun Sep 17 20:52:18 2006 From: pinterface at gmail.com (pinterface) Date: Sun, 17 Sep 2006 20:52:18 -0000 Subject: [elephant-devel] pthreads-related patch Message-ID: <016c01c6da9b$2afd0990$3100a8c0@trinity> Hello again. Unrelated to my previous patch, while finally loading elephant onto a threaded SBCL, I had some problems with compiling ele-bdb throwing me into SBCL's ldb debugger. Tracked it down, and it turns out the libpthread.so which elephant loads on linux is the culprit; I'm guessing it conflicts with the threading code SBCL is already using. But that's really beside the point: if *sleepycat-pthreads-path* is nil, elephant shouldn't try to load it. A patch is attached to fix just that. Whether this will cause me grief with multiple threads bouncing data around is yet to be seen, but at least none of the backend BDB tests fail. :) -pinterface (Progressing glacially.) -------------- next part -------------- A non-text attachment was scrubbed... Name: elephant-libpthreads.patch Type: application/octet-stream Size: 974 bytes Desc: not available URL: From eslick at csail.mit.edu Mon Sep 18 04:17:29 2006 From: eslick at csail.mit.edu (Ian Eslick) Date: Mon, 18 Sep 2006 00:17:29 -0400 Subject: [elephant-devel] pthreads-related patch In-Reply-To: <016c01c6da9b$2afd0990$3100a8c0@trinity> References: <016c01c6da9b$2afd0990$3100a8c0@trinity> Message-ID: <450E1DD9.7010408@csail.mit.edu> I think the pthreads library is required for BDB to run properly. This may be an interaction caused by the latest version of SBCL. I'll look at this from my end this week and promote your patch to HEAD if it is a solution to the general problem. Thank you! Ian pinterface wrote: > Hello again. > > Unrelated to my previous patch, while finally loading elephant onto a > threaded SBCL, I had some problems with compiling ele-bdb throwing me into > SBCL's ldb debugger. Tracked it down, and it turns out the libpthread.so > which elephant loads on linux is the culprit; I'm guessing it conflicts with > the threading code SBCL is already using. > > But that's really beside the point: if *sleepycat-pthreads-path* is nil, > elephant shouldn't try to load it. A patch is attached to fix just that. > Whether this will cause me grief with multiple threads bouncing data around > is yet to be seen, but at least none of the backend BDB tests fail. :) > > -pinterface > > (Progressing glacially.) > > ------------------------------------------------------------------------ > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From pinterface at gmail.com Wed Sep 20 01:07:31 2006 From: pinterface at gmail.com (pinterface) Date: Wed, 20 Sep 2006 01:07:31 -0000 Subject: [elephant-devel] pthreads-related patch References: <016c01c6da9b$2afd0990$3100a8c0@trinity> <450E1DD9.7010408@csail.mit.edu> Message-ID: <039401c6dc51$272ab5d0$3100a8c0@trinity> On 2006.09.18 04:17, "Ian Eslick" wrote: > I think the pthreads library is required for BDB to run properly. This > may be an interaction caused by the latest version of SBCL. I'll look > at this from my end this week and promote your patch to HEAD if it is a > solution to the general problem. I doubt my patch is a solution to whatever the underlying problem actually is, given that it doesn't change the default behavior under SBCL/Linux. But it is useful inasmuch as it lets me set *sleepycat-pthreads-path* to nil and continue development. Fixing the real problem would be even better, of course, so good luck with your investigation. :) -pinterface > Thank you! > Ian > > pinterface wrote: > > Hello again. > > > > Unrelated to my previous patch, while finally loading elephant onto a > > threaded SBCL, I had some problems with compiling ele-bdb throwing me into > > SBCL's ldb debugger. Tracked it down, and it turns out the libpthread.so > > which elephant loads on linux is the culprit; I'm guessing it conflicts with > > the threading code SBCL is already using. > > > > But that's really beside the point: if *sleepycat-pthreads-path* is nil, > > elephant shouldn't try to load it. A patch is attached to fix just that. > > Whether this will cause me grief with multiple threads bouncing data around > > is yet to be seen, but at least none of the backend BDB tests fail. :) > > > > -pinterface > > > > (Progressing glacially.) From pinterface at gmail.com Fri Sep 22 11:07:38 2006 From: pinterface at gmail.com (pinterface) Date: Fri, 22 Sep 2006 11:07:38 -0000 Subject: [elephant-devel] Goodbye make; asdf, you're on! Message-ID: <007401c6de37$524db630$3100a8c0@trinity> As I mentioned earlier on this list, I thought it would be nice to eliminate the need for a Makefile, and let asdf handle the compilation of C files as necessary, so the resulting .so files wind up in, and get loaded from, the same place as all the .fasl files, even under systems such as asdf-binary-locations and common-lisp-controller. The attached patch does just that. This patch also integrates my two previous patches, mostly because I forgot to start from a clean tree. Some notes, because it's a sizeable patch and notes are good: * All variables in config.lisp can be set between loading the elephant and ele-bdb systems. In my case, that means: (defmethod perform :after ((o load-op) (c (eql (find-system :elephant)))) (flet ((fs (s) (find-symbol (symbol-name s) :elephant))) (eval `(setf ,(fs :*sleepycat-foreign-library-path*) "/usr/lib/libdb-4.4.so" ,(fs :*sleepycat-pthreads-path*) nil)))) in the .asd file for the system which depends on elephant. Interactive use might be more along the lines of: (asdf:operate 'asdf:load-op :elephant) (setf elephant:*sleepycat-foreign-library-path* "/usr/lib/libdb-4.4.so") (asdf:operate 'asdf:load-op :ele-bdb) * Two new special variables are added: *sleepycat-include-dir* and *sleepycat-lib-dir*, which take the place of the Makefile's DBINCDIR and DBLIBDIR. They're easy to set (see above), but need smarter defaults so they don't have to be. * I took the liberty of moving uffi:load-foreign-library calls out of sleepycat.lisp and memutil.lisp and into (asdf:perform load-op), though how they wind up there may not be immediately obvious. Hopefully you can follow it. * There's groundwork to support compilers other than gcc. It may or may not be useful. * Much of the C-file groundwork could probably be factored into an asdf extension which other projects could benefit from. I've had enough fun for one day, so that can wait. :) -pinterface Was the code supposed to get shorter doing that? -------------- next part -------------- A non-text attachment was scrubbed... Name: elephant-no-make2.patch Type: application/octet-stream Size: 13444 bytes Desc: not available URL: