=== Cassandra consistency model

link:https://cassandra.apache.org/[Cassandra] is an
link:https://en.wikipedia.org/wiki/Eventual_consistency[eventually consistent] data store.
This means that replica can hold diverging data, but are guaranteed to converge over time.

Several mechanisms are built in Cassandra to enforce this convergence, and need to be
leveraged by *Distributed Server Administrator*. Namely
link:https://docs.datastax.com/en/dse/5.1/dse-admin/datastax_enterprise/tools/nodetool/toolsRepair.html[nodetool repair],
link:https://cassandra.apache.org/doc/latest/operating/hints.html[Hinted hand-off] and
link:https://cassandra.apache.org/doc/latest/operating/read_repair.html[Read repair].

The {server-name} tries to mitigate inconsistencies by relying on
link:https://docs.datastax.com/en/archived/cassandra/3.0/cassandra/dml/dmlConfigConsistency.html[QUORUM] read and write levels.
This means that a majority of replica are needed for read and write operations to be performed. This guaranty is needed
as the Mailbox is a complex datamodel with several layers of metadata, and needs "read-your-writes" guaranties that QUORUM
read+writes delivers.

Critical business operations, like UID allocation, rely on strong consistency mechanisms brought by
link:https://www.datastax.com/blog/2013/07/lightweight-transactions-cassandra-20[lightweight transaction].

==== About multi data-center setups

As strong consistency is required for some operation, especially regarding IMAP monotic UID and MODSEQ generation,
and as lightweight transactions are slow across data centers, running James with a
link:https://docs.datastax.com/en/ddac/doc/datastax_enterprise/production/DDACmultiDCperWorkloadType.html[multi data-center]
Cassandra setup is discouraged.

However, xref:{xref-base}/configure/cassandra.adoc[this page] enables setting alternative read level,
which could be acceptable regarding limited requirements. `LOCAL_QUORUM` coupled with `LOCAL_SERIAL`
is likely the only scalable setup. Some options were added to turn off SERIAL consistency usage for message
and mailbox management. However, the use of Lightweight Transaction cannot be disabled for UIDs and ModSeqs.

Running the {server-name} IMAP server in a multi datacenter setup will likely result either in data loss,
or very slow operations - as we rely on monotic UID generation, without strong consistency, UIDs could be allocated
several times.

We did wire a multi-DC friendly distributed, POP3 only server that leverages acceptable performance while staying
consistent. This is achieved by having a reduced feature set - supporting only the POP3 server and using messageIds as
identifiers (generated without synchronisation using TimeUUIDs). You can find this application
link:https://github.com/apache/james-project/tree/master/server/apps/distributed-pop3-app[on GitHub]. In the future,
JMAP support could be added, but requires followup developments as some components critically depends on UIDs
(for instance the search).
