= Distributed James Server &mdash; Operator guide
:navtitle: Operator guide

:xref-base: distributed
:mailet-repository-path-prefix: cassandra
:backend-name: cassandra
:sample-configuration-prefix-url: https://github.com/apache/james-project/blob/master/server/apps/distributed-app/sample-configuration
:server-name: Distributed James Server
include::partial$operate/guide.adoc[]

== Solving cassandra inconsistencies

Cassandra backend uses data duplication to workaround Cassandra query
limitations. However, Cassandra is not doing transaction when writing in
several tables, this can lead to consistency issues for a given piece of
data. The consequence could be that the data is in a transient state
(that should never appear outside of the system).

Because of the lack of transactions, it’s hard to prevent these kind of
issues. We had developed some features to fix some existing cassandra
inconsistency issues that had been reported to James.

=== Jmap message fast view projections

When you read a Jmap message, some calculated properties are expected to
be fast to retrieve, like `preview`, `hasAttachment`. James achieves it
by pre-calculating and storing them into a caching table
(`message_fast_view_projection`). Missing caches are populated on
message reads and will temporarily decrease the performance.

==== How to detect the outdated projections

You can watch the `MessageFastViewProjection` health check at
xref:{xref-base}/operate/webadmin.adoc#_check_all_components[webadmin documentation].
It provides a check based on the ratio of missed projection reads.

==== How to solve

Since the MessageFastViewProjection is self healing, you should be
concerned only if the health check still returns `degraded` for a while,
there’s a possible thing you can do is looking at James logs for more
clues.

=== Mailboxes

`mailboxPath` and `mailbox` tables share common fields like `mailboxId`
and mailbox `name`. A successful operation of creating/renaming/delete
mailboxes has to succeed at updating `mailboxPath` and `mailbox` table.
Any failure on creating/updating/delete records in `mailboxPath` or
`mailbox` can produce inconsistencies.

==== How to detect the inconsistencies

If you found the suspicious `MailboxNotFoundException` in your logs.
Currently, there’s no dedicated tool for that, we recommend scheduling
the SolveInconsistencies task below for the mailbox object on a regular
basis, avoiding peak traffic in order to address both inconsistencies
diagnostic and fixes.

==== How to solve

An admin can run offline webadmin
xref:{xref-base}/operate/webadmin.adoc#_fixing_mailboxes_inconsistencies[solve Cassandra mailbox object inconsistencies task]
in order to sanitize his
mailbox denormalization.

In order to ensure being offline, stop the traffic on SMTP, JMAP and
IMAP ports, for example via re-configuration or firewall rules.

=== Mailboxes Counters

James maintains a per mailbox projection for message count and unseen
message count. Failures during the denormalization process will lead to
incorrect results being returned.

==== How to detect the inconsistencies

Incorrect message count/message unseen count could be seen in the
`Mail User Agent` (IMAP or JMAP). Invalid values are reported in the
logs as warning with the following class
`org.apache.james.mailbox.model.MailboxCounters` and the following
message prefix: `Invalid mailbox counters`.

==== How to solve

Execute the
xref:{xref-base}/operate/webadmin.adoc#_recomputing_mailbox_counters[recompute Mailbox counters task].
This task is not concurrent-safe. Concurrent
increments & decrements will be ignored during a single mailbox
processing. Re-running this task may eventually return the correct
result.

=== Messages

Messages are denormalized and stored in both `imapUidTable` (source of
truth) and `messageIdTable`. Failure in the denormalization process will
cause inconsistencies between the two tables.

==== How to detect the inconsistencies

User can see a message in JMAP but not in IMAP, or mark a message as
`SEEN' in JMAP but the message flag is still unchanged in IMAP.

==== How to solve

Execute the
xref:{xref-base}/operate/webadmin.adoc#_fixing_message_inconsistencies[solve Cassandra message inconsistencies task]. This task is not
concurrent-safe. User actions concurrent to the inconsistency fixing
task could result in new inconsistencies being created. However the
source of truth `imapUidTable` will not be affected and thus re-running
this task may eventually fix all issues.

=== Quotas

User can monitor the amount of space and message count he is allowed to
use, and that he is effectively using. James relies on an event bus and
Cassandra to track the quota of an user. Upon Cassandra failure, this
value can be incorrect.

==== How to detect the inconsistencies

Incorrect quotas could be seen in the `Mail User Agent` (IMAP or JMAP).

==== How to solve

Execute the
xref:{xref-base}/operate/webadmin.adoc#_recomputing_current_quotas_for_users[recompute Quotas counters task]. This task is not concurrent-safe. Concurrent
operations will result in an invalid quota to be persisted. Re-running
this task may eventually return the correct result.

=== RRT (RecipientRewriteTable) mapping sources

`rrt` and `mappings_sources` tables store information about address
mappings. The source of truth is `rrt` and `mappings_sources` is the
projection table containing all mapping sources.

==== How to detect the inconsistencies

Right now there’s no tool for detecting that, we’re proposing a
https://issues.apache.org/jira/browse/JAMES-3069[development plan]. By
the mean time, the recommendation is to execute the
`SolveInconsistencies` task below in a regular basis.

==== How to solve

Execute the Cassandra mapping `SolveInconsistencies` task described in
xref:{xref-base}/operate/webadmin.adoc#_operations_on_mappings_sources[webadmin documentation]

== Setting Cassandra user permissions

When a Cassandra cluster is serving more than a James cluster, the
keyspaces need isolation. It can be achieved by configuring James server
with credentials preventing access or modification of other keyspaces.

We recommend you to not use the initial admin user of Cassandra and
provide a different one with a subset of permissions for each
application.

=== Prerequisites

We’re gonna use the Cassandra super users to create roles and grant
permissions for them. To do that, Cassandra requires you to login via
username/password authentication and enable granting in cassandra
configuration file.

For example:

....
echo -e "\nauthenticator: PasswordAuthenticator" >> /etc/cassandra/cassandra.yaml
echo -e "\nauthorizer: org.apache.cassandra.auth.CassandraAuthorizer" >> /etc/cassandra/cassandra.yaml
....

=== Prepare Cassandra roles & keyspaces for James

==== Create a role

Have a look at
http://cassandra.apache.org/doc/3.11.11/cql/security.html[cassandra documentation] section `CREATE ROLE` for more information

E.g.

....
CREATE ROLE james_one WITH PASSWORD = 'james_one' AND LOGIN = true;
....

==== Create a keyspace

Have a look at
http://cassandra.apache.org/doc/3.11.11/cql/ddl.html[cassandra documentation] section `CREATE KEYSPACE` for more information

==== Grant permissions on created keyspace to the role

The role to be used by James needs to have full rights on the keyspace
that James is using. Assuming the keyspace name is `james_one_keyspace`
and the role be `james_one`.

....
GRANT CREATE ON KEYSPACE james_one_keyspace TO james_one; // Permission to create tables on the appointed keyspace
GRANT SELECT ON KEYSPACE james_one_keyspace TO james_one; // Permission to select from tables on the appointed keyspace
GRANT MODIFY ON KEYSPACE james_one_keyspace TO james_one; // Permission to update data in tables on the appointed keyspace
....

*Warning*: The granted role doesn’t have the right to create keyspaces,
thus, if you haven’t created the keyspace, James server will fail to
start is expected.

*Tips*

Since all of Cassandra roles used by different James are supposed to
have a same set of permissions, you can reduce the works by creating a
base role set like `typical_james_role` with all of necessary
permissions. After that, with each James, create a new role and grant
the `typical_james_role` to the newly created one. Note that, once a
base role set is updated ( granting or revoking rights) all granted
roles are automatically updated.

E.g.

....
CREATE ROLE james1 WITH PASSWORD = 'james1' AND LOGIN = true;
GRANT typical_james_role TO james1;

CREATE ROLE james2 WITH PASSWORD = 'james2' AND LOGIN = true;
GRANT typical_james_role TO james2;
....

==== Revoke harmful permissions from the created role

We want a specific role that cannot describe or query the information of
other keyspaces or tables used by another application. By default,
Cassandra allows every role created to have the right to describe any
keyspace and table. There’s no configuration that can make effect on
that topic. Consequently, you have to accept that your data models are
still being exposed to anyone having credentials to Cassandra.

For more information, have a look at
http://cassandra.apache.org/doc/3.11.11/cql/security.html[cassandra documentation] section `REVOKE PERMISSION`.

Except for the case above, the permissions are not auto available for a
specific role unless they are granted by `GRANT` command. Therefore, if
you didn’t provide more permissions than
link:#_grant_permissions_on_created_keyspace_to_the_role[granting
section], there’s no need to revoke.

== Cassandra table level configuration

While _Distributed James_ is shipped with default table configuration
options, these settings should be refined depending of your usage.

These options are:

* The https://cassandra.apache.org/doc/latest/operating/compaction.html[compaction algorithms]
* The https://cassandra.apache.org/doc/latest/operating/bloom_filters.html[bloom filter sizing]
* The https://cassandra.apache.org/doc/latest/operating/compression.html?highlight=chunk%20size[chunk size]
* The https://www.datastax.com/blog/2011/04/maximizing-cache-benefit-cassandra[cachingoptions]

The compaction algorithms allow a tradeoff between background IO upon
writes and reads. We recommend:

* Using *Leveled Compaction Strategy* on
read intensive tables subject to updates. This limits the count of
SStables being read at the cost of more background IO. High garbage
collections can be caused by an inappropriate use of Leveled Compaction
Strategy.
* Otherwise use the default *Size Tiered Compaction Strategy*.

Bloom filters help avoiding unnecessary reads on SSTables. This
probabilistic data structure can tell an entry absence from a SSTable,
as well as the presence of an entry with an associated probability. If a
lot of false positives are noticed, the size of the bloom filters can be
increased.

As explained in
https://thelastpickle.com/blog/2018/08/08/compression_performance.html[this post],
chunk size used upon compression allows a tradeoff between reads
and writes. A smaller size will mean decreasing compression, thus it
increases data being stored on disk, but allow lower chunks to be read
to access data, and will favor reads. A bigger size will mean better
compression, thus writing less, but it might imply reading bigger
chunks.

Cassandra enables a key cache and a row cache. Key cache enables to skip
reading the partition index upon reads, thus performing 1 read to the
disk instead of 2. Enabling this cache is globally advised. Row cache
stores the entire row in memory. It can be seen as an optimization, but
it might actually use memory no longer available for instance for file
system cache. We recommend turning it off on modern SSD hardware.

A review of your usage can be conducted using
https://cassandra.apache.org/doc/latest/tools/nodetool/nodetool.html[nodetool]
utility. For example `nodetool tablestats \{keyspace\}` allows reviewing
the number of SSTables, the read/write ratios, bloom filter efficiency.
`nodetool tablehistograms \{keyspace\}.\{table\}` might give insight about
read/write performance.

Table level options can be changed using *ALTER TABLE* for example with
the https://cassandra.apache.org/doc/latest/tools/cqlsh.html[cqlsh]
utility. A full compaction might be needed in order for the changes to
be taken into account.

== Updating Cassandra schema version

A schema version indicates you which schema your James server is relying
on. The schema version number tracks if a migration is required. For
instance, when the latest schema version is 2, and the current schema
version is 1, you might think that you still have data in the deprecated
Message table in the database. Hence, you need to migrate these messages
into the MessageV2 table. Once done, you can safely bump the current
schema version to 2.

Relying on outdated schema version prevents you to benefit from the
newest performance and safety improvements. Otherwise, there’s something
very unexpected in the way we manage cassandra schema: we create new
tables without asking the admin about it. That means your James version
is always using the last tables but may also take into account the old
ones if the migration is not done yet.

=== How to detect when we should update Cassandra schema version

When you see in James logs
`org.apache.james.modules.mailbox.CassandraSchemaVersionStartUpCheck`
showing a warning like `Recommended version is versionX`, you should
perform an update of the Cassandra schema version.

Also, we keep track of changes needed when upgrading to a newer version.
You can read this
https://github.com/apache/james-project/blob/master/upgrade-instructions.md[upgrade
instructions].

=== How to update Cassandra schema version

These schema updates can be triggered by webadmin using the Cassandra
backend. Following steps are for updating Cassandra schema version:

* At the very first step, you need to
xref:{xref-base}/operate/webadmin.adoc#_retrieving_current_cassandra_schema_version[retrieve
current Cassandra schema version]
* And then, you
xref:{xref-base}/operate/webadmin.adoc#_retrieving_latest_available_cassandra_schema_version[retrieve
latest available Cassandra schema version] to make sure there is a
latest available version
* Eventually, you can update the current schema version to the one you
got with
xref:{xref-base}/operate/webadmin.adoc#_upgrading_to_the_latest_version[upgrading to
the latest version]

Otherwise, if you need to run the migrations to a specific version, you
can use
xref:{xref-base}/operate/webadmin.adoc#_upgrading_to_a_specific_version[Upgrading to a
specific version]