score:3
kafka is not really well-suited to anything that boils down to looking for a needle in a haystack, because all you can do is linear search.
getting rarely-more-than-once delivery in kafka is probably best accomplished by using an external datastore with much better query support (cassandra or elasticsearch are two of a great many options here). then you have the producer check if the message it wants to write is in that datastore, and you have a consumer of the topic whose sole purpose is to write the messages to that datastore. absent size-based retention and a keying scheme where different messages have the same keys, this should be fail-safe (meaning i can't think of how it's not): you won't erroneously decide not to write a message which you should write.
Source: stackoverflow.com
Related Query
- Kafka: Publish message only if it doesn't already exist
- SBT Publish only when version does not exist
- Is it possible to publish message to Kafka topics within Async calls like Future?
- Only first message in Kafka stream gets processed
- Scala Slick, how to create Schema ONLY if it does not exist
- sbt - exclude certain dependency only during publish
- Create Spark DataFrame in Spark Streaming from JSON Message on Kafka
- Akka Actor how to only process the latest message
- How to store only latest key values in a kafka topic
- Converting Case Classes with params as Case Classes to Avro Message to send to Kafka
- How to safely skip messages using Lagom Kafka Message Broker API?
- Spark Streaming + Kafka: how to check name of topic from kafka message
- Get kafka record timestamp from kafka message
- Akka: persist to Cassandra and publish to Kafka multiple events
- Spark JDBC - How to stop automatic creation of table if table doesnt exist
- Spark structured steaming from kafka - last message processed again after resume from checkpoint
- Why does auto-commit enabled Kafka client commit latest produced message's offset during consumer close even if the message was not consumed yet?
- Java/Scala Kafka Producer does not send message to topic
- Scala: Is hashCode() & equals() for case classes only generated if not already specified in a non-trivial superclass?
- Reading Message from Kafka Topic and Dump it into HDFS
- What is the way to only publish sbt documentation?
- Scala: Create object only if it doesn't exist yet
- How to parse Json formatted Kafka message in spark streaming
- Play Framework, Scala - sending websocket message to the client only if new data available
- How to consume only latest offset in Kafka topic
- Better way to create new key in map if key doesnt exist in Scala?
- Does the partition id of the Kafka Topic to which message was sent maches with partition id of open method in ForeachWriter?
- Get last inserted message from kafka topic
- How to read Avro Encoded kafka message in scala without knowing avro schema?
- kafka streams application only prints when application is killed
More Query from same tag
- Spark Graphx Vertices with Multiple Attributes (Type Mismatch - Product with Serializable Error)
- Developer guide for Kafka and Scala integration
- How to investigate differences in performance between logically equivalent Scala functions?
- Comparing string values of two columns in scala
- Why Parent class is allowing to Child class to make Parent class's methods as abstract in Child class?
- ColumnNotFound problem with Magic in play scala
- Scala map function to remove fields
- How to get a concatenated String of values from JSON response in Gatling?
- Sink inside Actor invoked during prod, but not during test
- Scala: what is a CompactBuffer?
- Scala placeholder syntax for anonymous function
- Cross-version suffixes in SBT with regards to scala-xml
- not found: value expectMsg
- How to cast an array of struct in a spark dataframe using selectExpr?
- Facing Error "Failed to load PDF document." in scala when send pdf file as bytearray in response (scala)
- How to match Dataframe column names to Scala case class attributes?
- Scalaz: combine Writer and State (and/or Lens)
- Does TVar in cats-stm require an immutable value?
- How to convert a sequence of Tuples into JSON array in spray
- Exception while parsing 7z files in java/scala
- How to get all sub-strings of a specific format from a string
- How to pass implicit parameter explicitly?
- akka-kryo-serialization ClassNotFoundException: scala.Enumeration$Val
- Dinamic case match in Scala based on fixed elements in a list
- Write code that shows the difference in salaries between the top-paid employee and others per department
- Spark - select multiple columns from array object
- Scala - How to iterate over tuples on RDD?
- Scala-cat's IOApp in an OSGi context
- Cost of Builder#result for immutable collections?
- Co- and contravariant types in generic priority queue