[Solved]-Cassandra big table migration bottleneck-scala

conn.withSessionDo executes custom CQL query using current shared connection to Cassandra:

Allows to use Cassandra Session in a safe way without risk of forgetting to close it. The Session object obtained through this method is a proxy to a shared, single Session associated with the cluster.

Internally, the shared underlying Session will be closed shortly after all the proxies are closed.

You can rewrite your code using saveToCassandra approach which is more typical.

As for my personal experience working with Spark+Cassandra, the slowest point for such queries is the Cassandra itself: data scans for huge tables are really slow (compared to Parquet).