score:3
Accepted answer
thanks for comments from harald and others, i found out a couple of things, which led me to achieve normal processing behaviour -
- test code with local mode, yarn isnt the biggest help in debugging
- for some reason, the process method of foreach sink doesnt allow calling other methods. when i put my business logic directly in there, it works.
hope it helps others.
Source: stackoverflow.com
Related Query
- Structured Streaming - Foreach Sink
- How to write JDBC Sink for Spark Structured Streaming [SparkException: Task not serializable]?
- Spark Structured Streaming Multiple WriteStreams to Same Sink
- spark structured streaming avro to avro and custom Sink
- Error in Spark Structured Streaming w/ File Source and File Sink
- Spark Structured Streaming MemoryStream report No data selected when use for Custom Sink
- No Parallelism with Custom Spark Structured Streaming Sink
- Spark Structured Streaming for appending to text file using foreach
- Spark Structured Streaming with HBase Sink
- Spark Structured Streaming with foreach
- Spark structured streaming - join static dataset with streaming dataset
- Monitoring Structured Streaming
- Spark Structured Streaming ForeachWriter and database performance
- Using Spark Structured Streaming with Trigger.Once
- How to use Spark Structured Streaming with Kafka Direct Stream?
- How to continuously monitor a directory by using Spark Structured Streaming
- Structured streaming with periodically updated static dataset
- Limit kafka batch size when using Spark Structured Streaming
- Combining/Updating Cassandra Queried data to Structured Streaming receieved from Kafka
- Simulate Lag Function - Spark structured streaming
- Watermarking for Spark structured streaming with three way joins
- Outer join two Datasets (not DataFrames) in Spark Structured Streaming
- Geting messages of Offset is getting reset in structured streaming mode in Spark
- How to use fully formed SQL with spark structured streaming
- Incompatible Jackson version: Spark Structured Streaming
- Spark Structured Streaming with Hbase integration
- Spark Structured Streaming recovering from a query exception
- Spark Structured Streaming MemoryStream + Row + Encoders issue
- Unbounded table is spark structured streaming
- How to use kafka.group.id and checkpoints in spark 3.0 structured streaming to continue to read from Kafka where it left off after restart?
More Query from same tag
- comparing Scala lists with Java lists
- How to pass Configuration object to controller in unit test
- Increasing scala/java heap size in scala shell script
- How do I serialize the value of a Map when running mkstring?
- Application crashes after upgrading to play 2.2
- How can I filter out just one ocurrence in a string vector in Scala?
- How to share a validation.Constraint across multiple Forms?
- Scala classNotFound on Any type
- Scala: Why is the PECS principle not applied to Function1?
- compilation nap - How can I make sbt to ring a bell when compilation is ready?
- Flink - importing types doesn't fix "could not find implicit value for evidence parameter of type ....TypeInformation"
- Getting the type of the default case in pattern maching
- Scala until loop with floating points
- Scala unpickling from string
- Does Call by Name or Call by Value have an effect in Big O Notation?
- SonarQube runner in sbt: findbugs, scalastyle
- How to pull all Int vals out of List[Any]?
- Why can't I seem to exclude this transitive dependency from build.sbt? (Continues to show up in ivyReport)
- Replicate values in a column using Scala/Spark
- Where did Option[T] come from in Scala?
- Compare two dataframes and update the values
- Scala Pattern match the result of a method during unit testing
- Spark broadcast serialization of custom BiMap class
- Storing thrift-serialized objects on disk
- Multiple pattern matches with same input variable
- Can a type conflict between typeclass evidence parameter and concrete instance parameter be resolved?
- How to register UDF to use in SQL and DataFrame?
- Using AWS S3 Java client to obtain directory and object metadata
- Comprehension of Actor with ExecutionContext
- Scala for loop Replace on List