From eslick at media.mit.edu Sat Mar 1 00:03:10 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Fri, 29 Feb 2008 19:03:10 -0500 Subject: [elephant-devel] Representational Question In-Reply-To: <1B1F38CE-9239-4083-80B6-685318384C28@infoway.net> References: <6A96C939-ECF2-4FC6-B12A-106B3AF64CE5@media.mit.edu> <1B1F38CE-9239-4083-80B6-685318384C28@infoway.net> Message-ID: <3D2937E6-8088-4073-8228-DAE01B9A17AF@media.mit.edu> On Feb 29, 2008, at 6:50 PM, lists at infoway.net wrote: > Hi Ian, > > Thanks for the prompt response. I know the querying facility is not > necessarily a priority at this time, but will someday become a > reality :) > > To tell you the truth, we haven't really had any direct experience > with Elephant in production or larger-scale type projects. However, > we do feel that the whole concept of object prevalence given the > complexity of the overall data model would make Elephant a more > appropriate framework than continuing the relational path (maybe > we're just wrong and Elephant is not best suited for this at all). > As it is, we currently need to do a lot of work to maintain all the > data relations and integrity in the current system and hopefully > working only with the object models would make things easier and > more "maintainable". Granted, I agree that at this moment, it's a > lot easier to formulate those queries in SQL, but I'd like to at > least be able to setup a parallel model and migrate data over so we > could compare performance (we're not even going to talk about the > complexity/difficulty of querying in Elephant, since we know that at > this stage, it is much more complex than SQL queries). > > I hope I'm not wrong, but definitely your opinion is worth more > since you (et al) know a lot more about this than us. Well, the first thing that occurs to me is search. I don't know what the performance would be, but you might just try a search-style algorithm. Use map-index: and for each element, just chase the dependency chain. (accessor (accessor (accessor object))) If you have a multi-valued slot, you'll have to expand the search for all children. There may be issues with this, not least of all of which is performance, but it's a good place to start. It would be interesting to compare the performance, look at different approaches and compare that to a pure SQL solution. > As for the second question, the answer is no. The objects would not > be stored in bulk. The idea is to keep an audit log of user- > initiated changes on individual entities (e.g. changing a Person's > address, or correcting a name, or assigning a health insurance plan, > etc). Hmmm...One way to do this is to use :after methods on the slot accessors you want to log. Those :after methods will write a log entry into the database and the log and the change will all be committed at the same time. You can arrange this so that all writes within a transaction gets written at once. You could also use :after methods on initialize-instance to catch object creation, etc. I haven't thought this through fully, but that's one way to do it. > Thanks, > Waldo > > On Feb 29, 2008, at 3:36 PM, Ian Eslick wrote: > >> Hi Waldo, >> >> Why do you want to migrate to Elephant for production and not stick >> with something like CL-SQL or cl-perec on top of a relational >> database so you get all the facilities that you're familiar with? >> >> Also, please don't expect a query system anytime soon. Finishing >> it is not in my critical path right now and no one else has stepped >> up and volunteered to lead or help with it. >> >> As for your query problem, I think the SQL solution for queries >> like that is likely to be faster in the end than putting this into >> Elephant. Elephant is not intended or designed to support >> efficient relational operations. That's what relational DBs are >> for! :) >> >> Wait until the next big update to elephant before you go too far >> down this road, I'm hoping that some new features I'm planning at >> least make this a little bit easier. >> >> For your second question, if you are going to save/store the >> objects in bulk, you can just use standard classes. Then you can >> have a transaction to fetch/diff/write the composite object to >> ensure atomicity of updates. This diff would also produce your >> log. However that means that you lose the indexing capability of >> persistent objects. >> >> Ian >> >> On Feb 29, 2008, at 2:47 PM, lists at infoway.net wrote: >> >>> Hi all, >>> >>> As I'm further exploring more and more things to do in Elephant >>> and Lisp, I think we're ready to start migrating some of our RoR >>> apps over, if not just as an exercise, we'll someday migrate them >>> to production. >>> >>> Since we all have a very strong and hard-headed background on >>> MySQL and relational models, it's been extremely difficult for us >>> to migrate away from that mentality and think of objects and some >>> of Elephant's terminology such as class indexes, which kind of >>> confuse us into thinking that a class index allows us to look at a >>> set of objects in a similar way as a MySQL table. >>> >>> I've read and seen in the src the beginning efforts to building a >>> query system into Elephant. That would be great and as our efforts >>> approach that phase, we hope to contribute to it. >>> >>> So, in this email, first I will ask for advise as to how to best >>> represent the structure of our objects/classes and indices in >>> Elephant in order to ultimately be able to query the data. Again, >>> I'm not going to ask for the querying strategy (just yet) but >>> ultimately, we will need to be able to answer queries like this. >>> Obviously I don't expect anyone to give me the full representation >>> of this, but any advise/hints as to best represent them will help >>> greatly. >>> >>> We have a database with many related tables. For simplicity >>> purposes, we'll describe a simplified scenario. We have a table >>> with people information (e.g. first,last names, date of birth, and >>> gender). We have a linked table with each person's addresses >>> (multiple addresses in case they moved. Each address is >>> timestamped so the most recent address is the current address). >>> Then, each person may be subscribed to one or more health >>> insurance plans, and so there is a table linking each person to >>> one or more health insurance plans (and a table that defines the >>> health insurance plans) >>> >>> Now, each person may select up to N preferred medical offices >>> where they would like to receive treatment. Again, there is a >>> table that links the person with one or more medical office. >>> Needless to say, there is a table of medical offices. Each medical >>> office is also linked to a timestamped address table, where the >>> most recent address is the current one (in the event the office >>> moves). To further expand on the issue, each office has one or >>> more doctors rendering services, so there is a table that links >>> the offices to the doctors, and of course, there is a table of >>> doctors that contains basic information, such as fname, lname, and >>> gender. Last, but not least, a doctor may be specialized in >>> multiple areas, so there is a table that links doctors to all the >>> specialties they have been certified on, and thus there is yet >>> another table that lists all possible specialties. >>> >>> Now, assuming I was able to explain the scenario correctly, we >>> then have users asking the system for information such as: >>> >>> "List all people (subscribers), who are male and live in zip code >>> 33012 who are contracted under Health Insurance Plan A that have >>> selected (as their preferred medical office) medical offices with >>> male cardiologists that work within 10 miles of 33012 zip code or >>> in MIAMI-DADE county and whose office names contain the sequence >>> of letters 'HEAL'" >>> >>> The way we see it, the concept of tables disappears and so do the >>> tables that provide many-to-many joins. So, we end up with some >>> classes such as "Person" which contains a reference to a list of >>> "Address" objects, and a list of preferred "Medical-Office" >>> objects, where each Medical-Office object has a list of Doctor >>> objects and each Doctor has a list of Specialty objects, etc, etc. >>> >>> Now, we assume that each of these classes will need to maintain >>> multiple indices, such as the Person class being index on first >>> name, last name, dob, gender, among others. The Address class >>> indexed on zip code, county name, among others, and so on and so >>> forth. >>> >>> The querying is one problem. The data representation is another. >>> We think it's clear that we should have, as an example, a Person >>> class. However, the representation of the links between a Person >>> and its Addresses or Medical-Offices is not 100% clear. If we >>> represent them as a slot in the Person class, where this slot >>> would be a List or a set of references to the Address class, then >>> in order for us to query on those, means that we always need to >>> fetch all objects in those slots in order to apply any search >>> criteria, which seems like a bottleneck. If that was the solution, >>> I assume we could implement logic such that Addresses are pushed >>> into the list, so that the most recent address is in the CAR, so >>> we wouldn't necessarily need to read the entire list of Addresses >>> for each member, but just fetch the CAR of the slot. >>> >>> Now, onto the second question. One of the other requirements we >>> have is that we need to keep an audit log of data changes. The way >>> we do it in RoR is relatively simple. We fetch an object from the >>> DB and present it on the browser. When the user submits, we fetch >>> another fresh copy from the DB and if the timestamps are the same >>> (meaning no one else changed the record) we compare changes to the >>> object's attributes (slots). If there are any differences, we save >>> the changes (we're trying to avoid unnecessary trips to the DB) >>> and if the changes are saved successfully, we write a log of ONLY >>> the attributes that were changed (which is pretty trivial in Ruby). >>> >>> From what we've read in Elephant's manual, this seems harder >>> because we don't want to work directly off the Elephant object but >>> a memory copy while the user takes his/her time in the browser and >>> after submitting, we would take the changes and commit them to the >>> Elephant object. Makes me think that we would need to classes for >>> each object (one with and one without the persistent metaclass). >>> The other problem would be how to "easily" have two objects >>> introspect themselves and spit out the slots that changed between >>> the two. >>> >>> Are we looking at this incorrectly? Any advise would be greatly >>> appreciated. >>> >>> Thanks, >>> Waldo >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From leslie.polzer at gmx.net Sat Mar 1 21:23:44 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sat, 1 Mar 2008 22:23:44 +0100 (CET) Subject: [elephant-devel] More from Postmodern Message-ID: <61027.88.73.228.137.1204406624.squirrel@mail.stardawn.org> I noticed two apparently minor errors with PM: First: 2008-03-01 19:47:18 CET ERROR: prepared statement "TREE32FIRST" already exists 2008-03-01 19:47:18 CET ERROR: current transaction is aborted, commands ignored until end of transaction block Second: 2008-03-01 20:42:35 CET ERROR: duplicate key violates unique constraint "slots_qi_key" 2008-03-01 20:42:35 CET CONTEXT: SQL statement "INSERT INTO slots (qi, value) VALUES ( $1 , $2 )" PL/pgSQL function "ins_upd_slots" line 9 at SQL statement Both of them might be the result of a race condition. Also, PM's persistent slot reader will raise an UNBOUND-SLOT exception instead of calling SLOT-UNBOUND. I believe the standard permits not calling SLOT-UNBOUND when a slot is not bound, but it's counter-intuitive anyway since that's the behaviour for STANDARD-CLASS and there doesn't really seem to be any reason not to emulate this behaviour. Leslie From leslie.polzer at gmx.net Sun Mar 2 09:22:12 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 2 Mar 2008 10:22:12 +0100 (CET) Subject: [elephant-devel] MOP and initforms Message-ID: <64176.88.73.202.143.1204449732.squirrel@mail.stardawn.org> It's annoying that the MOP doesn't honor initforms for persistent classes, resulting in unbound slots where none where planned. Is this intentional, and if yes, what's the rationale? Leslie From leslie.polzer at gmx.net Sun Mar 2 09:47:51 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 2 Mar 2008 10:47:51 +0100 (CET) Subject: [elephant-devel] Re: Lisp backend In-Reply-To: <871F6FD5-74AD-4479-94B3-5A211DE7F841@media.mit.edu> References: <871F6FD5-74AD-4479-94B3-5A211DE7F841@media.mit.edu> Message-ID: <61637.88.73.202.143.1204451271.squirrel@mail.stardawn.org> > The quad-tree is definitely not a B+Tree, nor one that solves any of > the issues of writing to disk. Well, so much for superficial research. Sorry. > It's easy enough to find a reference > implementation of a B+Tree and re-implement it; as long as we've > thought through all the tradeoffs (I think Henrik made this point). Speaking of which, I think the Right Thing is developing the B+Tree library separately from Elephant, so others will be able to use it and to achieve weak coupling. > Part of the challenge is being able to read/write and cache the > underlying disk storage. Do we write directly into binary pages in > memory, and read/write them, or do we manage all this in lisp and > serialize lisp BTree nodes? We can't serialize the entire BTree > whenever it changes, so we need to keep track of nodes and serialize > them whenever they change. However, nodes in lisp memory are going to > have variable size and we don't know how big they will be until they > are serialized. The tradeoffs here can get very interesting. Yes, let's gather questions like these as proposed on a central site. TC and BDB support variable-length values, so we can just look at their solution for this problem. > One of the biggest challenges is efficiently managing concurrency. If > you want multithreaded or multi-process operation, and you want > multiple transactions to proceed concurrently then you need some way > for all the threads of control to coordinate who is writing what data > so that transaction A doesn't commit data that transaction B depends > on, when transaction B is committing after A. The PostgreSQL manual has some interesting information on this topic. > Further, if you want to support multiple machines, then this pool of > locks needs to be distributed or exported via a server protocol. Let's stop here. We can think about this later. > One problem is that lisp doesn't provide native access to any locking > mechanism. I think it's OK to use uffi/cffi to talk to a standard C > library that does have these low level calls though. Yes. > The other approach to fine-grained locking is to use speculative > concurrency (roughly what Rucksack calls versioning). This means that > we assume there are no conflicts, don't bother to lock anything, copy- > on-writes and invalidate transactions at commit time that do have > conflicts (i.e. that read and operated on anything that was written). I believe the canonical term for this is "Optimistic Concurrency Control". Leslie From leslie.polzer at gmx.net Sun Mar 2 09:53:05 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 2 Mar 2008 10:53:05 +0100 (CET) Subject: [elephant-devel] Data mismatch Message-ID: <61903.88.73.202.143.1204451585.squirrel@mail.stardawn.org> The following just happened with DB-Postmodern: 1) Process A has been running for some time. 2) Process B was started. 3) Indexed class instances were added in B. 4) Those instances were visible in B, but not in A. 5) Both processes point to the same databases and are usually working correctly. 6) Restarting A's Lisp image resolved the issue, while reopening the store did not. So basically what happened is that something was really fishy, but Elephant didn't notice. Any idea on what might have happened? This could become serious if it happened again... Leslie From leslie.polzer at gmx.net Sun Mar 2 09:57:07 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 2 Mar 2008 10:57:07 +0100 (CET) Subject: [elephant-devel] Data mismatch In-Reply-To: <61903.88.73.202.143.1204451585.squirrel@mail.stardawn.org> References: <61903.88.73.202.143.1204451585.squirrel@mail.stardawn.org> Message-ID: <62100.88.73.202.143.1204451827.squirrel@mail.stardawn.org> PM caching was disabled all the time, by the way. From leslie.polzer at gmx.net Sun Mar 2 13:19:48 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Sun, 2 Mar 2008 14:19:48 +0100 (CET) Subject: [elephant-devel] Data mismatch Message-ID: <63006.88.73.202.143.1204463988.squirrel@mail.stardawn.org> > The following just happened with DB-Postmodern: Seems to be reproducible. I will check out whether this is resolved by Ian's multi-store indexed classes patch. Leslie From killerstorm at newmail.ru Sun Mar 2 14:57:32 2008 From: killerstorm at newmail.ru (Alex Mizrahi) Date: Sun, 2 Mar 2008 16:57:32 +0200 Subject: [elephant-devel] Re: Data mismatch References: <61903.88.73.202.143.1204451585.squirrel@mail.stardawn.org> Message-ID: LPP> 3) Indexed class instances were added in B. do you mean you've added instances with make-instance, or you did something with schema? LPP> 4) Those instances were visible in B, but not in A. you mean not accessible via get-instances-by-class? LPP> Any idea on what might have happened? just as idea, do you run your code in an explicit transaction? get-instances-by-class does SQL queries, so i see two possibilities: * new values were not seen due to transaction isolation (you didn't run fresh transaction, somehow) * it did _different_ queries in different processes. like process A created btree for class index, but process B did not notice this and made another one, overwritting A's entry in class-root table. if this happened due to race condition, it should be fixed when we'll move to serializable transaction isolation. otherwise, this could be a bug.. if this problem is reprodicble, please check oids of class index in different processes From eslick at media.mit.edu Sun Mar 2 22:05:54 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sun, 2 Mar 2008 17:05:54 -0500 Subject: [elephant-devel] MOP and initforms In-Reply-To: <64176.88.73.202.143.1204449732.squirrel@mail.stardawn.org> References: <64176.88.73.202.143.1204449732.squirrel@mail.stardawn.org> Message-ID: <0A333D95-DA15-4290-BEFC-B0789A4E4D89@media.mit.edu> Are you talking about class slots or instance slots? If instance slots, then this is a bug although it's surprising since it should be caught by the tests. Maybe this was introduced in the new inst init patch from a few weeks ago? Can you given an example of the failure condition? Ian On Mar 2, 2008, at 4:22 AM, Leslie P. Polzer wrote: > > It's annoying that the MOP doesn't honor initforms for persistent > classes, > resulting in unbound slots where none where planned. > > Is this intentional, and if yes, what's the rationale? > > Leslie > > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Mon Mar 3 01:05:16 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Sun, 2 Mar 2008 20:05:16 -0500 Subject: [elephant-devel] Re: Lisp backend In-Reply-To: <61637.88.73.202.143.1204451271.squirrel@mail.stardawn.org> References: <871F6FD5-74AD-4479-94B3-5A211DE7F841@media.mit.edu> <61637.88.73.202.143.1204451271.squirrel@mail.stardawn.org> Message-ID: >> It's easy enough to find a reference >> implementation of a B+Tree and re-implement it; as long as we've >> thought through all the tradeoffs (I think Henrik made this point). > > Speaking of which, I think the Right Thing is developing the B+Tree > library separately from Elephant, so others will be able to use it > and to achieve weak coupling. Definitely! It's also one good place to split the work up among several contributors so the whole project doesn't fall on one primary developer! >> The other approach to fine-grained locking is to use speculative >> concurrency (roughly what Rucksack calls versioning). This means >> that >> we assume there are no conflicts, don't bother to lock anything, >> copy- >> on-writes and invalidate transactions at commit time that do have >> conflicts (i.e. that read and operated on anything that was written). > > I believe the canonical term for this is "Optimistic Concurrency > Control". Quite right! We should move this discussion to the wiki and start capturing requirements, a basic decomposition, and some design decisions to make. See: http://trac.common-lisp.net/elephant/wiki/DesignTradeoffs http://trac.common-lisp.net/elephant/wiki/LispDataStore Ian From leslie.polzer at gmx.net Mon Mar 3 09:12:13 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 3 Mar 2008 10:12:13 +0100 (CET) Subject: [elephant-devel] MOP and initforms In-Reply-To: <0A333D95-DA15-4290-BEFC-B0789A4E4D89@media.mit.edu> References: <64176.88.73.202.143.1204449732.squirrel@mail.stardawn.org> <0A333D95-DA15-4290-BEFC-B0789A4E4D89@media.mit.edu> Message-ID: <63417.88.73.224.141.1204535533.squirrel@mail.stardawn.org> > Are you talking about class slots or instance slots? If instance > slots, then this is a bug although it's surprising since it should be > caught by the tests. Maybe this was introduced in the new inst init > patch from a few weeks ago? > > Can you given an example of the failure condition? I should have added that I'm talking about a schema evolution thing here, i.e. the situation concerns slots *added* to a class. The slots of instances of such a class are unbound instead of being assigned their initform, if any. Leslie From leslie.polzer at gmx.net Mon Mar 3 09:44:32 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 3 Mar 2008 10:44:32 +0100 (CET) Subject: [elephant-devel] Representational Question Message-ID: <65022.88.73.224.141.1204537472.squirrel@mail.stardawn.org> > Since we all have a very strong and hard-headed background on MySQL > and relational models, it's been extremely difficult for us to migrate > away from that mentality and think of objects and some of Elephant's > terminology such as class indexes, which kind of confuse us into > thinking that a class index allows us to look at a set of objects in a > similar way as a MySQL table. No. A class itself does that. The indexing mechanism is just a way of automatically storing every new instance and gathering those instances. A class slot index is actually pretty much the same as a column index in SQL. > I've read and seen in the src the beginning efforts to building a > query system into Elephant. That would be great and as our efforts > approach that phase, we hope to contribute to it. You can use MAP-* with appropriate functional queries until then. It's different from SQL queries. More than one leven of sorting would have to be implemented by you. > So, in this email, first I will ask for advise as to how to best > represent the structure of our objects/classes and indices in Elephant > in order to ultimately be able to query the data. Again, I'm not going > to ask for the querying strategy (just yet) but ultimately, we will > need to be able to answer queries like this. Obviously I don't expect > anyone to give me the full representation of this, but any advise/ > hints as to best represent them will help greatly. Looks like you could make use of some professional consulting here. > The way we see it, the concept of tables disappears As such yes. But you can model classes with tables and vice versa. > and so do the > tables that provide many-to-many joins. So, we end up with some > classes such as "Person" which contains a reference to a list of > "Address" objects, and a list of preferred "Medical-Office" objects, > where each Medical-Office object has a list of Doctor objects and each > Doctor has a list of Specialty objects, etc, etc. Yes. > Now, we assume that each of these classes will need to maintain > multiple indices, such as the Person class being index on first name, > last name, dob, gender, among others. The Address class indexed on zip > code, county name, among others, and so on and so forth. If you like it simple (and non-performant), you don't need any indices at all. Just use MAP-CLASS. > The querying is one problem. The data representation is another. We > think it's clear that we should have, as an example, a Person class. > However, the representation of the links between a Person and its > Addresses or Medical-Offices is not 100% clear. If we represent them > as a slot in the Person class, where this slot would be a List or a > set of references to the Address class, then in order for us to query > on those, means that we always need to fetch all objects in those > slots in order to apply any search criteria, which seems like a > bottleneck. If that was the solution, I assume we could implement > logic such that Addresses are pushed into the list, so that the most > recent address is in the CAR, so we wouldn't necessarily need to read > the entire list of Addresses for each member, but just fetch the CAR > of the slot. Now here you should use btree indices [1]. > Now, onto the second question. Yuk. Seems there's a bunch of separate issues in this "second question". I'll try to handle them separately. > One of the other requirements we have > is that we need to keep an audit log of data changes. You can have a simple log with some trivial MOP changes. > From what we've read in Elephant's manual, this seems harder because > we don't want to work directly off the Elephant object but a memory > copy while the user takes his/her time in the browser and after > submitting, we would take the changes and commit them to the Elephant > object. Sounds sensible, yes. > Makes me think that we would need to classes for each object > (one with and one without the persistent metaclass). No, not necessarily. New instances only get added to the persistent storage when they are of indexed classes. Otherwise you need an explicit ADD-TO-ROOT. If you need both indexing and this fine-grained behaviour, use your own indexing stuff with non-indexed objects. > The other problem > would be how to "easily" have two objects introspect themselves and > spit out the slots that changed between the two. Use the MOP facilities. An alternative solution to the whole problem is off-loading this to the client. Have JavaScript record which fields changed. Leslie > Are we looking at this incorrectly? Any advise would be greatly > appreciated. You seem to be a bit confused. Try hands-on experiments. :) Leslie [1] http://common-lisp.net/project/elephant/doc/BTree-Indexing.html#BTree-Indexing From leslie.polzer at gmx.net Mon Mar 3 10:00:29 2008 From: leslie.polzer at gmx.net (Leslie P. Polzer) Date: Mon, 3 Mar 2008 11:00:29 +0100 (CET) Subject: [elephant-devel] Converting MIGRATE to BACKUP Message-ID: <61936.88.73.224.141.1204538429.squirrel@mail.stardawn.org> To be able to backup all objects to another store, would it be enough to modify MIGRATE so it doesn't update the home store of objects? Are there any subtle gotchas? Leslie From eslick at media.mit.edu Mon Mar 3 12:41:24 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Mon, 3 Mar 2008 07:41:24 -0500 Subject: [elephant-devel] Representational Question In-Reply-To: <65022.88.73.224.141.1204537472.squirrel@mail.stardawn.org> References: <65022.88.73.224.141.1204537472.squirrel@mail.stardawn.org> Message-ID: > > No, not necessarily. New instances only get added to the persistent > storage when they are of indexed classes. Otherwise you need > an explicit ADD-TO-ROOT. To clarify, the slot values are immediately stored to the database, but if you don't add the object to the root, then it becomes 'unreachable' unless you've enabled class indices, as Leslie says above. From henrik at evahjelte.com Mon Mar 3 13:06:12 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Mon, 3 Mar 2008 14:06:12 +0100 Subject: [elephant-devel] Converting MIGRATE to BACKUP In-Reply-To: <61936.88.73.224.141.1204538429.squirrel@mail.stardawn.org> References: <61936.88.73.224.141.1204538429.squirrel@mail.stardawn.org> Message-ID: <50e8e4f60803030506h6c6990ebtb97f79603c5ac101@mail.gmail.com> On Mon, Mar 3, 2008 at 11:00 AM, Leslie P. Polzer wrote: > > To be able to backup all objects to another store, would it be > enough to modify MIGRATE so it doesn't update the home store > of objects? I've mentioned it before, but here is another ad: I have made a basic import/export facility that can dump and restore elephant databases to a file in sexp format. I use it to migrate data and to do backups. It feels safe that the data is in a human readable file. Currrent main restriction is that data should fit in main memory, but I will fix that sometime. It is hidden under the grand-prix project on common-lisp-net: common-lisp.net/project/grand-prix/darcs/gp-export /Henrik From eslick at media.mit.edu Mon Mar 3 13:14:49 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Mon, 3 Mar 2008 08:14:49 -0500 Subject: [elephant-devel] Converting MIGRATE to BACKUP In-Reply-To: <50e8e4f60803030506h6c6990ebtb97f79603c5ac101@mail.gmail.com> References: <61936.88.73.224.141.1204538429.squirrel@mail.stardawn.org> <50e8e4f60803030506h6c6990ebtb97f79603c5ac101@mail.gmail.com> Message-ID: <3919EC83-D52D-46C4-A38A-6E9F07A55C88@media.mit.edu> This would be great to integrate into the core code base! Do you think that would be reasonably easy to do? On Mar 3, 2008, at 8:06 AM, Henrik Hjelte wrote: > On Mon, Mar 3, 2008 at 11:00 AM, Leslie P. Polzer > wrote: >> >> To be able to backup all objects to another store, would it be >> enough to modify MIGRATE so it doesn't update the home store >> of objects? > > I've mentioned it before, but here is another ad: I have made a basic > import/export facility that can dump and restore elephant databases to > a file in sexp format. I use it to migrate data and to do backups. It > feels safe that the data is in a human readable file. > Currrent main restriction is that data should fit in main memory, but > I will fix that sometime. > > It is hidden under the grand-prix project on common-lisp-net: > common-lisp.net/project/grand-prix/darcs/gp-export > > /Henrik > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From henrik at evahjelte.com Mon Mar 3 13:33:22 2008 From: henrik at evahjelte.com (Henrik Hjelte) Date: Mon, 3 Mar 2008 14:33:22 +0100 Subject: [elephant-devel] Converting MIGRATE to BACKUP In-Reply-To: <3919EC83-D52D-46C4-A38A-6E9F07A55C88@media.mit.edu> References: <61936.88.73.224.141.1204538429.squirrel@mail.stardawn.org> <50e8e4f60803030506h6c6990ebtb97f79603c5ac101@mail.gmail.com> <3919EC83-D52D-46C4-A38A-6E9F07A55C88@media.mit.edu> Message-ID: <50e8e4f60803030533h4123a445x58ced9a3c1b406b9@mail.gmail.com> On Mon, Mar 3, 2008 at 2:14 PM, Ian Eslick wrote: > This would be great to integrate into the core code base! Do you > think that would be reasonably easy to do? yes. It adds some new dependencies, and it is not finished on the feature list yet, but sure. I actually intended it to be useful for other solutions as well (rucksack, acache etc.), and to be useful for migrating between these. But that is no reason it can't be under the elephant project. /Henrik > > > On Mar 3, 2008, at 8:06 AM, Henrik Hjelte wrote: > > > On Mon, Mar 3, 2008 at 11:00 AM, Leslie P. Polzer > > wrote: > >> > >> To be able to backup all objects to another store, would it be > >> enough to modify MIGRATE so it doesn't update the home store > >> of objects? > > > > I've mentioned it before, but here is another ad: I have made a basic > > import/export facility that can dump and restore elephant databases to > > a file in sexp format. I use it to migrate data and to do backups. It > > feels safe that the data is in a human readable file. > > Currrent main restriction is that data should fit in main memory, but > > I will fix that sometime. > > > > It is hidden under the grand-prix project on common-lisp-net: > > common-lisp.net/project/grand-prix/darcs/gp-export > > > > /Henrik > > > > _______________________________________________ > > elephant-devel site list > > elephant-devel at common-lisp.net > > http://common-lisp.net/mailman/listinfo/elephant-devel > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel > -- Henrik Hjelte henrik.hjelte at stix.to +46703993945 http://stix.to L?stmakarg 18-20 (IQube) Box 7438 S-103 91 Stockholm Sweden From lists at infoway.net Mon Mar 3 14:52:12 2008 From: lists at infoway.net (lists at infoway.net) Date: Mon, 3 Mar 2008 09:52:12 -0500 Subject: [elephant-devel] Representational Question In-Reply-To: <65022.88.73.224.141.1204537472.squirrel@mail.stardawn.org> References: <65022.88.73.224.141.1204537472.squirrel@mail.stardawn.org> Message-ID: <0AA328E1-9AE4-42F7-AF8D-99B0D6A75FF6@infoway.net> Leslie, Thanks for your answers and the clarifications. I think you're right in that we may need some consulting here. However, we're trying really hard to get into this and become proficient at it. As such, we would probably be interested in some consulting to help us get started so as to give us still a challenge to move forward. Sort of like kicking us off in the right direction. Since this is not a requirements for our client but more of an internal desire to move in this direction, we don't really have a budget (or remunerable need) to do this. Nonetheless, I'd like to discuss this offline with whomever is interested in lending a hand (for a fee of course) at this stage, and potentially be available as we move along the project. Thanks, Waldo On Mar 3, 2008, at 4:44 AM, Leslie P. Polzer wrote: > >> Since we all have a very strong and hard-headed background on MySQL >> and relational models, it's been extremely difficult for us to >> migrate >> away from that mentality and think of objects and some of Elephant's >> terminology such as class indexes, which kind of confuse us into >> thinking that a class index allows us to look at a set of objects >> in a >> similar way as a MySQL table. > > No. A class itself does that. The indexing mechanism is just a way > of automatically storing every new instance and gathering those > instances. > > A class slot index is actually pretty much the same as a column index > in SQL. > > >> I've read and seen in the src the beginning efforts to building a >> query system into Elephant. That would be great and as our efforts >> approach that phase, we hope to contribute to it. > > You can use MAP-* with appropriate functional queries until then. > It's different from SQL queries. More than one leven of sorting > would have to be implemented by you. > > >> So, in this email, first I will ask for advise as to how to best >> represent the structure of our objects/classes and indices in >> Elephant >> in order to ultimately be able to query the data. Again, I'm not >> going >> to ask for the querying strategy (just yet) but ultimately, we will >> need to be able to answer queries like this. Obviously I don't expect >> anyone to give me the full representation of this, but any advise/ >> hints as to best represent them will help greatly. > > Looks like you could make use of some professional consulting here. > > >> The way we see it, the concept of tables disappears > > As such yes. But you can model classes with tables and vice versa. > > >> and so do the >> tables that provide many-to-many joins. So, we end up with some >> classes such as "Person" which contains a reference to a list of >> "Address" objects, and a list of preferred "Medical-Office" objects, >> where each Medical-Office object has a list of Doctor objects and >> each >> Doctor has a list of Specialty objects, etc, etc. > > Yes. > > >> Now, we assume that each of these classes will need to maintain >> multiple indices, such as the Person class being index on first name, >> last name, dob, gender, among others. The Address class indexed on >> zip >> code, county name, among others, and so on and so forth. > > If you like it simple (and non-performant), you don't need any indices > at all. Just use MAP-CLASS. > > >> The querying is one problem. The data representation is another. We >> think it's clear that we should have, as an example, a Person class. >> However, the representation of the links between a Person and its >> Addresses or Medical-Offices is not 100% clear. If we represent them >> as a slot in the Person class, where this slot would be a List or a >> set of references to the Address class, then in order for us to query >> on those, means that we always need to fetch all objects in those >> slots in order to apply any search criteria, which seems like a >> bottleneck. If that was the solution, I assume we could implement >> logic such that Addresses are pushed into the list, so that the most >> recent address is in the CAR, so we wouldn't necessarily need to read >> the entire list of Addresses for each member, but just fetch the CAR >> of the slot. > > Now here you should use btree indices [1]. > > >> Now, onto the second question. > > Yuk. Seems there's a bunch of separate issues in this "second > question". > I'll try to handle them separately. > > >> One of the other requirements we have >> is that we need to keep an audit log of data changes. > > You can have a simple log with some trivial MOP changes. > > >> From what we've read in Elephant's manual, this seems harder because >> we don't want to work directly off the Elephant object but a memory >> copy while the user takes his/her time in the browser and after >> submitting, we would take the changes and commit them to the Elephant >> object. > > Sounds sensible, yes. > > >> Makes me think that we would need to classes for each object >> (one with and one without the persistent metaclass). > > No, not necessarily. New instances only get added to the persistent > storage when they are of indexed classes. Otherwise you need > an explicit ADD-TO-ROOT. > > If you need both indexing and this fine-grained behaviour, > use your own indexing stuff with non-indexed objects. > > >> The other problem >> would be how to "easily" have two objects introspect themselves and >> spit out the slots that changed between the two. > > Use the MOP facilities. > > An alternative solution to the whole problem is off-loading this > to the client. Have JavaScript record which fields changed. > > Leslie > > >> Are we looking at this incorrectly? Any advise would be greatly >> appreciated. > > You seem to be a bit confused. Try hands-on experiments. > > :) Leslie > > > [1] http://common-lisp.net/project/elephant/doc/BTree-Indexing.html#BTree-Indexing > > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From eslick at media.mit.edu Mon Mar 3 14:59:19 2008 From: eslick at media.mit.edu (Ian Eslick) Date: Mon, 3 Mar 2008 09:59:19 -0500 Subject: [elephant-devel] Converting MIGRATE to BACKUP In-Reply-To: <50e8e4f60803030533h4123a445x58ced9a3c1b406b9@mail.gmail.com> References: <61936.88.73.224.141.1204538429.squirrel@mail.stardawn.org> <50e8e4f60803030506h6c6990ebtb97f79603c5ac101@mail.gmail.com> <3919EC83-D52D-46C4-A38A-6E9F07A55C88@media.mit.edu> <50e8e4f60803030533h4123a445x58ced9a3c1b406b9@mail.gmail.com> Message-ID: It might make sense to just have it be a separate common-lisp.net project. We can have an optional elephant module that adds the appropriate dependencies and wraps it into the Elephant framework. (Perhaps a contrib for now until the features stabilize?) Ian On Mar 3, 2008, at 8:33 AM, Henrik Hjelte wrote: > On Mon, Mar 3, 2008 at 2:14 PM, Ian Eslick > wrote: >> This would be great to integrate into the core code base! Do you >> think that would be reasonably easy to do? > yes. It adds some new dependencies, and it is not finished on the > feature list yet, but sure. > > I actually intended it to be useful for other solutions as well > (rucksack, acache etc.), and to be useful for migrating between these. > But that is no reason it can't be under the elephant project. > > /Henrik > >> >> >> On Mar 3, 2008, at 8:06 AM, Henrik Hjelte wrote: >> >>> On Mon, Mar 3, 2008 at 11:00 AM, Leslie P. Polzer >>> wrote: >>>> >>>> To be able to backup all objects to another store, would it be >>>> enough to modify MIGRATE so it doesn't update the home store >>>> of objects? >>> >>> I've mentioned it before, but here is another ad: I have made a >>> basic >>> import/export facility that can dump and restore elephant >>> databases to >>> a file in sexp format. I use it to migrate data and to do backups. >>> It >>> feels safe that the data is in a human readable file. >>> Currrent main restriction is that data should fit in main memory, >>> but >>> I will fix that sometime. >>> >>> It is hidden under the grand-prix project on common-lisp-net: >>> common-lisp.net/project/grand-prix/darcs/gp-export >>> >>> /Henrik >> >> >>> _______________________________________________ >>> elephant-devel site list >>> elephant-devel at common-lisp.net >>> http://common-lisp.net/mailman/listinfo/elephant-devel >> >> _______________________________________________ >> elephant-devel site list >> elephant-devel at common-lisp.net >> http://common-lisp.net/mailman/listinfo/elephant-devel >> > > > > -- > Henrik Hjelte > henrik.hjelte at stix.to > +46703993945 > http://stix.to > L?stmakarg 18-20 (IQube) > Box 7438 > S-103 91 Stockholm > Sweden > _______________________________________________ > elephant-devel site list > elephant-devel at common-lisp.net > http://common-lisp.net/mailman/listinfo/elephant-devel From franks-muc at web.de Mon Mar 3 18:15:15 2008 From: franks-muc at web.de (Frank Schorr) Date: Mon, 03 Mar 2008 19:15:15 +0100 Subject: [elephant-devel] ELEPHANT-DESERIALIZATION-ERROR Message-ID: <999211748@web.de> This is on LispWorks/windows. I did an update today (darcs get http://www.common-lisp.net/project/elephant/darcs/elephant) and can't run the application which used to work in the past. A minor thing: (deftest base-string ... Error while reading: Saw character U+0080 with syntax type invalid. Solved this by commenting out ;; (in-out-equal (make-array 3 :initial-element #\?€™ :fill-pointer 2 ;; :element-type 'character)) ; test non-simple Unicode string, where supported But then I rezeived a ELEPHANT-DESERIALIZATION-ERROR. Backtrace is shown below. What can I do ? Any help will be very welcome. Best regards Frank Schorr This is my-config.lisp ((:compiler . :cygwin) (:berkeley-db-version . "4.5") (:berkeley-db-include-dir . "C:/Programme/Oracle/Berkeley DB 4.5.20/include/") (:berkeley-db-lib-dir . "C:/Programme/Oracle/Berkeley DB 4.5.20/bin/") (:berkeley-db-lib . "C:/Programme/Oracle/Berkeley DB 4.5.20/bin/libdb45.dll") (:berkeley-db-deadlock . "C:/Programme/Oracle/Berkeley DB 4.5.20/bin/db_deadlock.exe") ;; (:pthread-lib . nil) (:berkeley-db-cachesize . 20971520) (:berkeley-db-map-degree2 . t) (:clsql-lib-paths . nil) (:prebuilt-libraries . nil)) This the backtrace The condition # occurred 1 (continue) Rerun the test # 2 Signal an exceptional test failure and abort the test #. 3 Ignore the rest of the tests and explain current results 4 (abort) Abort job 6 :(BIND-STANDARD-STREAMS-AND-EXECUTE # 21CE43DF> (FUNCALL-BACKGROUND-JOB-AUX # BACKGROUND-REGION-EVAL (# # # # T NIL NIL))) Type :b for backtrace, :c