[elephant-devel] Garbage collection problem
Ian Eslick
eslick at csail.mit.edu
Thu Sep 27 13:58:50 UTC 2007
This was (sb-sprof:with-profiling (:mode :alloc ...) ...) ??
On Sep 27, 2007, at 9:39 AM, Chris Laux wrote:
>> How about just giving us a profile of memory usage?
>
> The profile is at the bottom of this message, in sbcl's sb-sprof
> format.
> If you can't interpret the numbers let me know and I'll dig up the
> explanation from the web. I'm not really sure how to interpret the
> results, and it might be buggy, because asking the profiler to report
> causes the same (similar?) condition as garbage collection, that is
> the
> system hangs for minutes then works on as if nothing had happened
> (added
> later: or never recovers again...)
>
>
>> A likely candidate is the serializer, but I'm sure there could be
>> other
>> corner cases.
>
> It occurred to me that I should just describe what I'm doing, maybe
> there's an obvious stupidity on my side. I'm indexing short
> messages to
> be queries by a text search engine.
>
> A main BTree maps from words to (sub-)BTrees that in turn map from
> ranking (e.g. time, for recency) to (persistent) result object. To
> answer a query, I then look up the (sub-)BTrees for each word in the
> query and map over their elements until I have enough to fill a result
> page with the intersection of their results.
>
> This is probably a rather extreme use of the DB, for example
> creating so
> many persistent instances from the store might be the problem for the
> GC? Is there a "nicer" way of doing something like this? E.g. mapping
> only to id's, so as not to reconstruct an instance if all I'm going to
> do with is is "intersect" it's occurrance between lists of messages?
>
> Cheers,
>
> Chris
>
>
> [NB: the order of the two output tables is reversed for ease of use]
>
>
> Number of samples: 390
> Sample interval: 4 regions (approximately 32 kB)
> Total sampling amount: 1560 regions (approximately 12480 kB)
> Number of cycles: 0
>
>
> Self Total Cumul
> Nr Count % Count % Count % Function
> ----------------------------------------------------------------------
> --
> 1 81 20.8 81 20.8 81 20.8
> ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM
> 2 73 18.7 73 18.7 154 39.5 (SB-PCL::FAST-METHOD
> ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>")
> "#<...>"
> . "#<...>"))
> 3 57 14.6 57 14.6 211 54.1
> ELEPHANT::DESERIALIZE-DATABASE-VERSION-VALUE
> 4 56 14.4 56 14.4 267 68.5
> ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM
> 5 50 12.8 50 12.8 317 81.3 FIND-PACKAGE
> 6 35 9.0 35 9.0 352 90.3
> ELEPHANT-SERIALIZER2::GET-CIRCULARITY-VECTOR
> 7 16 4.1 16 4.1 368 94.4
> ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR
> 8 12 3.1 12 3.1 380 97.4 ELEPHANT::GET-CACHED-
> INSTANCE
> 9 6 1.5 6 1.5 386 99.0 (LAMBDA (K V))
> 10 1 0.3 1 0.3 387 99.2 LIST
> 11 1 0.3 1 0.3 388 99.5
> ELEPHANT::MAKE-TRANSACTION-RECORD
> 12 1 0.3 1 0.3 389 99.7 (SB-PCL::FAST-METHOD
> MAKE-CURSOR (DB-BDB::BDB-BTREE))
> 13 1 0.3 1 0.3 390 100.0
> SB-KERNEL::%MAKE-INSTANCE-WITH-LAYOUT
> 14 0 0.0 244 62.6 390 100.0 (LABELS
> ELEPHANT-SERIALIZER2::%DESERIALIZE)
> 15 0 0.0 307 78.7 390 100.0
> ELEPHANT-SERIALIZER2::DESERIALIZE
> 16 0 0.0 378 96.9 390 100.0 (SB-PCL::FAST-METHOD
> CURSOR-PREV (DB-BDB::BDB-CURSOR))
> 17 0 0.0 387 99.2 390 100.0 "Unknown component:
> #x100F827230"
> 18 0 0.0 387 99.2 390 100.0 (LAMBDA NIL)
> 19 0 0.0 390 100.0 390 100.0 (SB-PCL::FAST-METHOD
> ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" .
> "#<...>"))
> 20 0 0.0 390 100.0 390 100.0 (SB-PCL::FAST-METHOD
> MAP-BTREE (T "#<...>" . "#<...>"))
> 21 0 0.0 390 100.0 390 100.0 GET-BTREE-SUBSEQ
> 22 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL
> ("#<...>"
> . "#<...>")))
> 23 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL
> ("#<...>"
> . "#<...>")))
> 24 0 0.0 390 100.0 390 100.0 (LAMBDA (&OPTIONAL
> ("#<...>"
> . "#<...>") "#<...>" . "#<...>"))
> 25 0 0.0 390 100.0 390 100.0 ANSWER-QUERY
> 26 0 0.0 390 100.0 390 100.0 PRODUCE-FEED
> 27 0 0.0 390 100.0 390 100.0 HUNCHENTOOT::PROCESS-
> REQUEST
> 28 0 0.0 390 100.0 390 100.0 HUNCHENTOOT::PROCESS-
> CONNECTION
> 29 0 0.0 390 100.0 390 100.0 "Unknown component:
> #x1000677630"
> 30 0 0.0 390 100.0 390 100.0 "foreign function
> call_into_lisp"
> 31 0 0.0 390 100.0 390 100.0 "foreign function
> new_thread_trampoline"
> 32 0 0.0 29 7.4 390 100.0 SB-INT:%FIND-PACKAGE-OR-
> LOSE
> 33 0 0.0 29 7.4 390 100.0
> SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE
> 34 0 0.0 29 7.4 390 100.0 INTERN
> 35 0 0.0 121 31.0 390 100.0 (SB-PCL::FAST-METHOD
> ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER))
> 36 0 0.0 121 31.0 390 100.0 (SB-PCL::FAST-METHOD
> ELEPHANT::DATABASE-VERSION :AROUND (T))
> 37 0 0.0 34 8.7 390 100.0 (FLET #:CLEANUP-FUN-279)
> 38 0 0.0 22 5.6 390 100.0 (FLET #:CLEANUP-FUN-1505)
> 39 0 0.0 21 5.4 390 100.0 TRANSLATE-AND-INTERN-
> SYMBOL
> 40 0 0.0 1 0.3 390 100.0 PACKAGE-NAME
> 41 0 0.0 2 0.5 390 100.0 (SB-PCL::FAST-METHOD
> CURSOR-LAST (DB-BDB::BDB-CURSOR))
> 42 0 0.0 1 0.3 390 100.0
> SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE
> ----------------------------------------------------------------------
> --
>
> Callers
> Total. Function
> Count % Count % Callees
> ----------------------------------------------------------------------
> --
> 30 7.7 (SB-PCL::FAST-METHOD
> ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35]
> 51 13.1 (SB-PCL::FAST-METHOD CURSOR-PREV
> (DB-BDB::BDB-CURSOR)) [16]
> 81 20.8 81 20.8 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM [1]
> ----------------------------------------------------------------------
> --
> 73 18.7 (LABELS
> ELEPHANT-SERIALIZER2::%DESERIALIZE) [14]
> 73 18.7 73 18.7 (SB-PCL::FAST-METHOD
> ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>")
> "#<...>"
> . "#<...>")) [2]
> ----------------------------------------------------------------------
> --
> 57 14.6 (SB-PCL::FAST-METHOD
> ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35]
> 57 14.6 57 14.6 ELEPHANT::DESERIALIZE-DATABASE-VERSION-
> VALUE [3]
> ----------------------------------------------------------------------
> --
> 22 5.6 (FLET #:CLEANUP-FUN-1505) [38]
> 34 8.7 (FLET #:CLEANUP-FUN-279) [37]
> 56 14.4 56 14.4 ELEPHANT-MEMUTIL::RETURN-BUFFER-STREAM [4]
> ----------------------------------------------------------------------
> --
> 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39]
> 29 7.4 SB-INT:%FIND-PACKAGE-OR-LOSE [32]
> 50 12.8 50 12.8 FIND-PACKAGE [5]
> ----------------------------------------------------------------------
> --
> 35 9.0 ELEPHANT-SERIALIZER2::DESERIALIZE [15]
> 35 9.0 35 9.0 ELEPHANT-SERIALIZER2::GET-CIRCULARITY-
> VECTOR [6]
> ----------------------------------------------------------------------
> --
> 16 4.1 ELEPHANT-SERIALIZER2::DESERIALIZE [15]
> 16 4.1 16 4.1
> ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR [7]
> ----------------------------------------------------------------------
> --
> 12 3.1 ELEPHANT-SERIALIZER2::DESERIALIZE [15]
> 12 3.1 12 3.1 ELEPHANT::GET-CACHED-INSTANCE [8]
> ----------------------------------------------------------------------
> --
> 6 1.5 "Unknown component: #x100F827230" [17]
> 6 1.5 6 1.5 (LAMBDA (K V)) [9]
> ----------------------------------------------------------------------
> --
> 1 0.3 (SB-PCL::FAST-METHOD
> ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" .
> "#<...>")) [19]
> 1 0.3 1 0.3 LIST [10]
> ----------------------------------------------------------------------
> --
> 1 0.3 (SB-PCL::FAST-METHOD
> ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" .
> "#<...>")) [19]
> 1 0.3 1 0.3 ELEPHANT::MAKE-TRANSACTION-RECORD [11]
> ----------------------------------------------------------------------
> --
> 1 0.3 "Unknown component: #x100F827230" [17]
> 1 0.3 1 0.3 (SB-PCL::FAST-METHOD MAKE-CURSOR
> (DB-BDB::BDB-BTREE)) [12]
> ----------------------------------------------------------------------
> --
> 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE
> [42]
> 1 0.3 1 0.3 SB-KERNEL::%MAKE-INSTANCE-WITH-LAYOUT [13]
> ----------------------------------------------------------------------
> --
> 244 62.6 ELEPHANT-SERIALIZER2::DESERIALIZE [15]
> 1 0.3 (LABELS
> ELEPHANT-SERIALIZER2::%DESERIALIZE) [14]
> 0 0.0 244 62.6 (LABELS ELEPHANT-SERIALIZER2::%
> DESERIALIZE) [14]
> 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39]
> 121 31.0 (SB-PCL::FAST-METHOD
> ELEPHANT::DATABASE-VERSION :AROUND (T)) [36]
> 29 7.4 INTERN [34]
> 1 0.3 (LABELS
> ELEPHANT-SERIALIZER2::%DESERIALIZE) [14]
> 73 18.7 (SB-PCL::FAST-METHOD
> ELEPHANT-SERIALIZER2::DESERIALIZE-STRING (("#<...>" . "#<...>")
> "#<...>"
> . "#<...>")) [2]
> ----------------------------------------------------------------------
> --
> 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST
> (DB-BDB::BDB-CURSOR)) [41]
> 305 78.2 (SB-PCL::FAST-METHOD CURSOR-PREV
> (DB-BDB::BDB-CURSOR)) [16]
> 0 0.0 307 78.7 ELEPHANT-SERIALIZER2::DESERIALIZE [15]
> 16 4.1
> ELEPHANT-SERIALIZER2::RELEASE-CIRCULARITY-VECTOR [7]
> 12 3.1 ELEPHANT::GET-CACHED-INSTANCE [8]
> 35 9.0
> ELEPHANT-SERIALIZER2::GET-CIRCULARITY-VECTOR [6]
> 244 62.6 (LABELS
> ELEPHANT-SERIALIZER2::%DESERIALIZE) [14]
> ----------------------------------------------------------------------
> --
> 378 96.9 "Unknown component: #x100F827230" [17]
> 0 0.0 378 96.9 (SB-PCL::FAST-METHOD CURSOR-PREV
> (DB-BDB::BDB-CURSOR)) [16]
> 22 5.6 (FLET #:CLEANUP-FUN-1505) [38]
> 51 13.1 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM
> [1]
> 305 78.2 ELEPHANT-SERIALIZER2::DESERIALIZE [15]
> ----------------------------------------------------------------------
> --
> 387 99.2 (LAMBDA NIL) [18]
> 0 0.0 387 99.2 "Unknown component: #x100F827230" [17]
> 1 0.3 (SB-PCL::FAST-METHOD MAKE-CURSOR
> (DB-BDB::BDB-BTREE)) [12]
> 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST
> (DB-BDB::BDB-CURSOR)) [41]
> 6 1.5 (LAMBDA (K V)) [9]
> 378 96.9 (SB-PCL::FAST-METHOD CURSOR-PREV
> (DB-BDB::BDB-CURSOR)) [16]
> ----------------------------------------------------------------------
> --
> 387 99.2 (SB-PCL::FAST-METHOD
> ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" .
> "#<...>")) [19]
> 0 0.0 387 99.2 (LAMBDA NIL) [18]
> 387 99.2 "Unknown component: #x100F827230" [17]
> ----------------------------------------------------------------------
> --
> 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T
> "#<...>" . "#<...>")) [20]
> 0 0.0 390 100.0 (SB-PCL::FAST-METHOD
> ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" .
> "#<...>")) [19]
> 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE
> [42]
> 1 0.3 ELEPHANT::MAKE-TRANSACTION-RECORD [11]
> 1 0.3 LIST [10]
> 387 99.2 (LAMBDA NIL) [18]
> ----------------------------------------------------------------------
> --
> 390 100.0 GET-BTREE-SUBSEQ [21]
> 0 0.0 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T
> "#<...>" .
> "#<...>")) [20]
> 390 100.0 (SB-PCL::FAST-METHOD
> ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" .
> "#<...>")) [19]
> ----------------------------------------------------------------------
> --
> 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" .
> "#<...>"))) [22]
> 0 0.0 390 100.0 GET-BTREE-SUBSEQ [21]
> 390 100.0 (SB-PCL::FAST-METHOD MAP-BTREE (T
> "#<...>" . "#<...>")) [20]
> ----------------------------------------------------------------------
> --
> 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" .
> "#<...>"))) [23]
> 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" .
> "#<...>"))) [22]
> 390 100.0 GET-BTREE-SUBSEQ [21]
> ----------------------------------------------------------------------
> --
> 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" .
> "#<...>")
> "#<...>" . "#<...>")) [24]
> 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" .
> "#<...>"))) [23]
> 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" .
> "#<...>"))) [22]
> ----------------------------------------------------------------------
> --
> 390 100.0 ANSWER-QUERY [25]
> 0 0.0 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" . "#<...>")
> "#<...>" . "#<...>")) [24]
> 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" .
> "#<...>"))) [23]
> ----------------------------------------------------------------------
> --
> 390 100.0 PRODUCE-FEED [26]
> 0 0.0 390 100.0 ANSWER-QUERY [25]
> 390 100.0 (LAMBDA (&OPTIONAL ("#<...>" .
> "#<...>")
> "#<...>" . "#<...>")) [24]
> ----------------------------------------------------------------------
> --
> 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27]
> 0 0.0 390 100.0 PRODUCE-FEED [26]
> 390 100.0 ANSWER-QUERY [25]
> ----------------------------------------------------------------------
> --
> 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28]
> 0 0.0 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27]
> 390 100.0 PRODUCE-FEED [26]
> ----------------------------------------------------------------------
> --
> 390 100.0 "Unknown component: #x1000677630" [29]
> 0 0.0 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28]
> 390 100.0 HUNCHENTOOT::PROCESS-REQUEST [27]
> ----------------------------------------------------------------------
> --
> 390 100.0 "foreign function call_into_lisp" [30]
> 0 0.0 390 100.0 "Unknown component: #x1000677630" [29]
> 390 100.0 HUNCHENTOOT::PROCESS-CONNECTION [28]
> ----------------------------------------------------------------------
> --
> 390 100.0 "foreign function
> new_thread_trampoline" [31]
> 0 0.0 390 100.0 "foreign function call_into_lisp" [30]
> 390 100.0 "Unknown component: #x1000677630" [29]
> ----------------------------------------------------------------------
> --
> 0 0.0 390 100.0 "foreign function
> new_thread_trampoline" [31]
> 390 100.0 "foreign function call_into_lisp" [30]
> ----------------------------------------------------------------------
> --
> 1 0.3 PACKAGE-NAME [40]
> 28 7.2 SB-INT:FIND-UNDELETED-PACKAGE-OR-
> LOSE [33]
> 0 0.0 29 7.4 SB-INT:%FIND-PACKAGE-OR-LOSE [32]
> 29 7.4 FIND-PACKAGE [5]
> ----------------------------------------------------------------------
> --
> 29 7.4 INTERN [34]
> 0 0.0 29 7.4 SB-INT:FIND-UNDELETED-PACKAGE-OR-LOSE [33]
> 1 0.3 PACKAGE-NAME [40]
> 28 7.2 SB-INT:%FIND-PACKAGE-OR-LOSE [32]
> ----------------------------------------------------------------------
> --
> 29 7.4 (LABELS
> ELEPHANT-SERIALIZER2::%DESERIALIZE) [14]
> 0 0.0 29 7.4 INTERN [34]
> 29 7.4 SB-INT:FIND-UNDELETED-PACKAGE-OR-
> LOSE [33]
> ----------------------------------------------------------------------
> --
> 121 31.0 (SB-PCL::FAST-METHOD
> ELEPHANT::DATABASE-VERSION :AROUND (T)) [36]
> 0 0.0 121 31.0 (SB-PCL::FAST-METHOD
> ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35]
> 30 7.7 ELEPHANT-MEMUTIL::GRAB-BUFFER-STREAM
> [1]
> 34 8.7 (FLET #:CLEANUP-FUN-279) [37]
> 57 14.6
> ELEPHANT::DESERIALIZE-DATABASE-VERSION-VALUE [3]
> ----------------------------------------------------------------------
> --
> 121 31.0 (LABELS
> ELEPHANT-SERIALIZER2::%DESERIALIZE) [14]
> 0 0.0 121 31.0 (SB-PCL::FAST-METHOD
> ELEPHANT::DATABASE-VERSION :AROUND (T)) [36]
> 121 31.0 (SB-PCL::FAST-METHOD
> ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35]
> ----------------------------------------------------------------------
> --
> 34 8.7 (SB-PCL::FAST-METHOD
> ELEPHANT::DATABASE-VERSION (DB-BDB::BDB-STORE-CONTROLLER)) [35]
> 0 0.0 34 8.7 (FLET #:CLEANUP-FUN-279) [37]
> 34 8.7 ELEPHANT-MEMUTIL::RETURN-BUFFER-
> STREAM [4]
> ----------------------------------------------------------------------
> --
> 22 5.6 (SB-PCL::FAST-METHOD CURSOR-PREV
> (DB-BDB::BDB-CURSOR)) [16]
> 0 0.0 22 5.6 (FLET #:CLEANUP-FUN-1505) [38]
> 22 5.6 ELEPHANT-MEMUTIL::RETURN-BUFFER-
> STREAM [4]
> ----------------------------------------------------------------------
> --
> 21 5.4 (LABELS
> ELEPHANT-SERIALIZER2::%DESERIALIZE) [14]
> 0 0.0 21 5.4 TRANSLATE-AND-INTERN-SYMBOL [39]
> 21 5.4 FIND-PACKAGE [5]
> ----------------------------------------------------------------------
> --
> 1 0.3 SB-INT:FIND-UNDELETED-PACKAGE-OR-
> LOSE [33]
> 0 0.0 1 0.3 PACKAGE-NAME [40]
> 1 0.3 SB-INT:%FIND-PACKAGE-OR-LOSE [32]
> ----------------------------------------------------------------------
> --
> 2 0.5 "Unknown component: #x100F827230" [17]
> 0 0.0 2 0.5 (SB-PCL::FAST-METHOD CURSOR-LAST
> (DB-BDB::BDB-CURSOR)) [41]
> 2 0.5 ELEPHANT-SERIALIZER2::DESERIALIZE [15]
> ----------------------------------------------------------------------
> --
> 1 0.3 (SB-PCL::FAST-METHOD
> ELEPHANT::EXECUTE-TRANSACTION (DB-BDB::BDB-STORE-CONTROLLER "#<...>" .
> "#<...>")) [19]
> 0 0.0 1 0.3 SB-ALIEN-INTERNALS:MAKE-ALIEN-VALUE [42]
> 1 0.3 SB-KERNEL::%MAKE-INSTANCE-WITH-
> LAYOUT [13]
> ----------------------------------------------------------------------
> --
> _______________________________________________
> elephant-devel site list
> elephant-devel at common-lisp.net
> http://common-lisp.net/mailman/listinfo/elephant-devel
More information about the elephant-devel
mailing list