score:3
the problem is that the anonymous function passed to mapcont
closes over next
. in turn, this is closed-over by the lazy variable we pass to enumiterator, which is closed-over by the new enumerator
formed by enumiterator1
, which is closed-over by the anonymous function in apply
, which is finally closed-over by the anonymous function passed to mapcont
for the next iteration.
so, by a chain of closures, each enumerator closes over its predecessor. this would probably happen whether next
was captured or not, so you'd have a minor memory leak either way. however, you end up capturing next
in one of these closures, which means that every value generated by your iterator stays in memory until the whole process is complete (and these values take up a lot of memory).
by moving next
inside the anonymous function passed to mapcont
, next
is not captured in our chain of closures any more, so the main memory leak disappears (although your closures still close over each other, which may be a concern).
the best way to fix this is probably to simplify it. as brian kernighan famously said:
everyone knows that debugging is twice as hard as writing a program in the first place. so if you're as clever as you can be when you write it, how will you ever debug it?
i'm not certain i fully understand the code, but i suspect the following is equivalent:
def enumiterator1[e, f[_]: monad](x: => iterator[e]) : enumeratort[e, f] =
new enumeratort[e, f] {
def apply[a] = {
val xs = x
def innerapply(s: stept[e, f, a]): iterateet[e, f, a] = {
if(xs.isempty) s.pointi
else {
val next = xs.next
s mapcont { cont => // renamed k to cont, as the function, rather than the variable, is k
cont(iteratee.elinput(next)) >>== innerapply
}
}
}
innerapply
}
}
you might also benefit from making things more explicit. for example, what if rather than having an anonymous enumeratort
that implicitly closes over anything it needs within its scope, you define a named class, with top level scope, and pass in anything it needs explicitly.
i used -xx:+heapdumponoutofmemoryerror
, visualvm, and javap
to find the cause of the issue. they should be everything you need.
update
i think i'm starting to grok what the code's supposed to do, and i've updated my code accordingly. i think the problem was the use of enumiterator1[e, f](xs).apply[a]
. the code was creating a new enumeratort
just to get at its apply method, but creating a by-name variable and closing over everything-and-its-dog in the process. since the value of xs
doesn't change from one recursion to the next, we create an innerapply
method which closes over the val xs
, and re-use innerapply
.
update 2
i was curious, so i had a look around in the scalaz source to see how they solve this problem. here's some code with a similar bent from scalaz itself:
def enumiterator[e, f[_]](x: => iterator[e])(implicit mo: monadpartialorder[f, io]) : enumeratort[e, f] =
new enumeratort[e, f] {
import mo._ // remove this line, and you can copy and paste it into your code
def apply[a] = {
def go(xs: iterator[e])(s: stept[e, f, a]): iterateet[e, f, a] =
if(xs.isempty) s.pointi
else {
s mapcont { k =>
val next = xs.next
k(elinput(next)) >>== go(xs)
}
}
go(x)
}
}
they use currying, rather than closure, to capture xs
, but it's still an "inner apply" approach.
Source: stackoverflow.com
Related Query
- Why does this Scalaz 7 enumerator leak memory?
- Why does a small change to this Scala code make such a huge difference to performance?
- Forward References - why does this code compile?
- Why does Scalaz use complex symbols and no in-code documentation?
- Why does this compile under Java 7 but not under Java 8?
- Why does this explicit call of a Scala method allow it to be implicitly resolved?
- Why does this Scala function compile when the argument does not conform to the type constraint?
- Scala - why Double consume less memory than Floats in this case?
- Why does Scalaz show up in my project's API docs?
- Why does this not give a type error?
- Why does this flatMap fail?
- Why does the Option's orNull method have this superfluous implicit argument?
- Why does the addition of enclosing parentheses change the result of this Scala expression?
- Why does Scala fail to find a secondary implicit value in this one particular case?
- Why does this Scala line return a Unit?
- Why does scala (1 to 1000).foreach throw an exception in this case?
- Why does this Scala for expression using tuples fail to compile?
- Why does this Scala code throw IllegalAccessError at runtime?
- Why does a variance annotation cause this subtyping relation not to be inferred by Scala?
- Why scala does not unify this type lambda with underlying type?
- Why does this partial application not compile?
- Why does this list-of-futures to future-of-list transformation compile and work?
- Why does this code typecheck in Scala 2.11 and what can I do about it?
- Why does this ScalaQuery statement only delete the odd rows?
- Why does this code need an empty line or a semicolon?
- Why does this getOrElse statement return type ANY?
- Why does this simple implicit stringToInt function cause a stack overflow?
- Why does this Iterator infinitely loop?
- Why this structural type bound does not work as expected?
- why does this scala by-name parameter behave weirdly
More Query from same tag
- Any ways to have generic case classes or have a trait that specifies a copy method with a specific param?
- Is there any way to get Key/Value type from a map in Scala?
- Merging two JSON Trees using Jerkson in Scala
- Scala how to update values in immutable list
- Can I read only wanted Files from a directory in scala-spark
- Scala replace " by \"
- Why does the compiler complain when using the abstract modifier in a trait in two cases?
- How to bind a array form field with Playframework scala
- How to fill missing values in SataFrame?
- Scala SBT fails - error while loading CharSequence (Scala 2.12.1, SBT 0.13.13)
- Adding task dependencies in .sbt file
- scala speed when using get() method on hash tables? (are temporary Option() objects generated?)
- Converting YAML to/from JSON using MoultingYaml
- How to define a function in scala for flatMap
- Return a List from multiple WS calls in a Scala foreach
- Is there a way to reference a function parameter that is shadowed by an inline class field?
- How to cast java interface to java object in scala
- Spark: understanding partitioning - cores
- Multilabel Classification using NaiveBayes Classifier in Spark
- Cannot generate indexable for a type containing a Map with a custom Key type
- Distinguishing between nothing and A=>Any for overloaded methods
- Return few variables of a class in scala
- Which union is more efficient: List / HashSet
- How to print all the columns of a Matrix
- Parsing Form Url Encoded Request To Case Class
- how to apply joins in spark scala when we have multiple values in the join column
- In Scala 2.10 or higher, how does one supply an "empty" catch block
- Is it possible to pass in a Play! Template as a variable?
- What is the best way to get the paths of all the files located in a GCS bucket from Spark in Scala?
- Reducing a List of Case Classes to a Count of the Case Classes