score:3

kafka is not really well-suited to anything that boils down to looking for a needle in a haystack, because all you can do is linear search.

getting rarely-more-than-once delivery in kafka is probably best accomplished by using an external datastore with much better query support (cassandra or elasticsearch are two of a great many options here). then you have the producer check if the message it wants to write is in that datastore, and you have a consumer of the topic whose sole purpose is to write the messages to that datastore. absent size-based retention and a keying scheme where different messages have the same keys, this should be fail-safe (meaning i can't think of how it's not): you won't erroneously decide not to write a message which you should write.


Related Query

More Query from same tag