# Upgrade instructions

This document covers every changes an Ops needs to be aware of when running James.

The following procedures are to take as it, and the Apache Software Foundation, nor its contributors, can not be 
responsible for any damages generated by following the below procedures.

Before performing these operations, you should ensure to have the skills to conduct the operations, and you should read other
software documentation. Do not follow this guide blindly!

## Unreleased

No specific operation to conduct from a 3.9.0 version.

## 3.9.0 version (tagged but not yet released)

Note: this section is in progress. It will be updated during all the development process until the release.
No changes yet.

Changes to apply between 3.8.x and 3.9.0 will be reported here.

Change list:

 - [JAMES-4046 Refactor and update apache-james-mailbox-lucene](#james-4046-refactor-and-update-apache-james-mailbox-lucene)
 - [Imap Packages](#imap-packages)
 - [Jmap uploads](#jmap-uploads)
 - [Mutualize quota table](#mutualize-quota-table)
 - [Drop Legacy Cassandra migrations](#drop-legacy-cassandra-migrations)
 - [Improve CassandraThreadIdGuessingAlgorithm](#improve-cassandrathreadidguessingalgorithm)
 - [Set up TTL on the mailbox_change and email_change tables](#set-up-ttl-on-the-mailboxchange-and-emailchange-tables)
 - [Change compaction strategy of blob_cache table](#change-compaction-strategy-of-blobcache-table)
 - [RRT forwards now rewrite senders](#rrt-forwards-now-rewrite-senders)
 - [Increase RabbitMQ consumer timeout on the task queue](#increase-rabbitmq-consumer-timeout-on-the-task-queue)
 - [JMX authentication for Spring](#jmx-authentication-for-spring)
 - [Java 21](#java-21)
 - [javax -> jakarta](#javax---jakarta)
 - [Make all queues on RabbitMQ quorum queue when `quorum.queues.enable=true`](#make-all-queues-on-rabbitmq-quorum-queue-when-quorumqueuesenabletrue)
 - [Migrate RabbitMQ classic queues to version 2](#migrate-rabbitmq-classic-queues-to-version-2)
 - [JAMES-3946 White list removals](#james-3946-white-list-removals)
 - [JAMES-4052 Details in quota index](#james-4052-details-in-quota-index)
 - [JAMES-1409 Change JPARecipientRewriteTable to store separate record per target address](#james-1409-change-jparecipientrewritetable-to-store-separate-record-per-target-address)
 - [JAMES-4118 Cleanup message previews](#james-4118-cleanup-message-previews)
 - [JAMES-4128 Breaking Mailet API changes](#james-4128-breaking-mailet-api-changes)

### JAMES-4128 Breaking Mailet API changes
Date: 02/04/2025

JIRA: https://issues.apache.org/jira/browse/JAMES-4126

The Mailet API has been slightly reworked to improve mailet encapsulation and make implementation easier. 
    
- Method getMailetConfig() has been removed
- Method getName() has been added with a default to the simple class name
- implementers no longer have to override getMailetInfo, init or destroy
- implementers of mailets can inject the configuration through the constructor when using guice. Constructor injection of the config is not compatible with spring applications.

### JAMES-4118 Cleanup message previews

Date: 12/03/2025

JIRA: https://issues.apache.org/jira/browse/JAMES-4118

Because message previews were not deleted upon mailbox deletion, a message previews cleanup extension (https://github.com/apache/james-project/pull/2667) has been made to remove redundant previews. Follow the instruction in README.md in the pull request to use this extension.  

### JAMES-1409 Change JPARecipientRewriteTable to store separate record per target address

Date: 09/10/2024

JIRA: https://issues.apache.org/jira/browse/JAMES-1409

The JPARecipientRewriteTable was modified to store multiple mappings of a single source as separate database rows,
each with a single target address, instead of a single row with a long semicolon-delimited multi-value target address.
This solves both the limitation on the number of mappings (imposed by the column maximum length), and the broken query
by target address.

For JPA users, the database schema update and data migration can be performed as follows:

- On the old James server (before the version upgrade), export all mappings using the webadmin
  interface (GET /mappings) and save the JSON result.
- Shut down the server.
- Drop the old table using any database administration tool (DROP TABLE JAMES_RECIPIENT_REWRITE).
- Upgrade and start the new James server. A new table will be created automatically with the new schema.
- Import the saved JSON mappings via the webadmin interface (PUT /mappings).

### JAMES-4052 Details in quota index and mailbox user

Date: 23/07/2024

JIRA: https://issues.apache.org/jira/browse/JAMES-4052

In order to build a comprehensive view for quotas using OpenSearch dashboard, the quota document indexed was enriched 
to include quota details.

As the OpenSearch mapping was changed, one can either:

 - Add the missing fields:

```
curl -X PUT \
  http://ip:port/quota_ratio_v1/_mapping \
  -H 'Content-Type: application/json' \
  -d "{
	\"properties\": {
		\"sizeUsed\": {
			\"type\": \"long\",
		},
		\"countUsed\": {
			\"type\": \"long\",
		},
		\"sizeLimit\": {
			\"type\": \"long\",
		},
		\"countLimit\": {
			\"type\": \"long\",
		},
		\"date\": {
			\"type\": \"date\",
			\"format\": \"uuuu-MM-dd'T'HH:mm:ssX||uuuu-MM-dd'T'HH:mm:ssXXX||uuuu-MM-dd'T'HH:mm:ssXXXXX\"
		}
	}
}"
```

 - Or consider this non-critical data and just start with a fresh index, which could be achieved with the following configuration:

```
opensearch.index.quota.ratio.name=james-quota-ratio-v2
opensearch.alias.read.quota.ratio.name=james-quota-ratio-read-v2
opensearch.alias.write.quota.ratio.name=james-quota-ratio-write-v2
```

We also added an optional field to the mailbox mapping for `user`. It allows doing per user analysis in OpenSearch dashboards.

In order to activate that field, include in `opensearch.properties`: 

```
opensearch.indexUser=true
```

And create the field:

```
curl -X PUT \
  http://ip:port/mailbox_v2/_mapping \
  -H 'Content-Type: application/json' \
  -d '{
    "properties": {
        "user": {
            "type": "keyword"
        }
    }
}'
```

### JAMES-4046 Refactor and update apache-james-mailbox-lucene

Date: 16/08/2024

JIRA: https://issues.apache.org/jira/browse/JAMES-4046

Lucene was upgraded from version 3 to version 9. Automatic upgrade is not supported with such a big upgrade thus it's required to delete/drop Lucene (file) index altogether and recreate it. 

Index recreation can be done with [ReIndexing all mails REST API](https://james.apache.org/server/manage-webadmin.html#ReIndexing_all_mails)

### JAMES-3946 White list removals

Date: 14/06/2024

JIRA: https://issues.apache.org/jira/browse/JAMES-3946

The experimental classes pertaining to "white list" management have been removed.

This includes:
- AbstractSQLWhitelistMatcher, IsInWhiteList, WhiteListManager, NetworkIsInWhitelist. Please use IsInDropList matcher instead.

### Migrate RabbitMQ classic queues to version 2

Date: 14/05/2024

It is recommended by RabbitMQ to upgrade the classic queues to version 2 for better performance: https://www.rabbitmq.com/blog/2023/05/17/rabbitmq-3.12-performance-improvements#classic-queues-massively-improved-classic-queues-v2-cqv2.

Existing version 1 classic queues would need to be deleted and let James re-create them as version 2.

Notice that to use classic queues version 2, you need at least RabbitMQ 3.10.0. If you want to stick with the older RabbitMQ 
versions and avoid this breaking change, you could set the JVM property `fallback.classic.queues.v1` to `true` (defaults to `false`).

Note that this fallback option would not work with RabbitMQ 4.0+ versions, as RabbitMQ 4.0 removed support for classic queues version 1.

If you upgrade your RabbitMQ cluster to RabbitMQ 4.0+, it would automatically convert classic queues version 1 to version 2 upon the upgrade, so no need to delete them.

### Change cluster.enabled in redis.properties to redis.topology

Date: 05/09/2024

Concerned products: Distributed James, Cassandra James

JIRA: https://issues.apache.org/jira/browse/JAMES-3693

Now James supports more than two topologies (previously there were just cluster and standalone).

Use `redis.topology` property instead.

### Make all queues on RabbitMQ quorum queue when `quorum.queues.enable=true`

Date: 16/04/2024

JIRA: https://issues.apache.org/jira/browse/JAMES-4027

Now `quorum.queues.enable=true` enforces all the queues used by James to be quorum queue to achieve fully high availability.

Some old classic queues (e.g. JMAP and mailbox eventbus work queues) would need to be deleted and let James re-create them as quorum queues.

### Java 21

Date: 25/03/2024

Apache James now requires the use of Java 21 and of maven 3.8.1+.

Extension developer might need to upgrade their extensions as well.

### javax -> jakarta

Date: 25/03/2024

Apache James migrated to jakarta as a replacement for javax.

Extension developers needs to also rely on Jakarta. This is mostly an import change:

```java
import javax.xxx;
```

Would become:


```java
import jakarta.xxx;
```

### JMX authentication for Spring

Date: 19/12/2023

Spring distribution now requires `jmxremote.access` and `jmxremote.password` files within the `/conf` folder in order to start.

### Increase RabbitMQ consumer timeout on the task queue

Date: 11/20/2023

JIRA: https://issues.apache.org/jira/browse/JAMES-3955

Before, the RabbitMQ default acknowledgement timeout on the `taskManagerWorkQueue` was 30 minutes.
This is not enough for long-running tasks, therefore we advise to increase the acknowledgement timeout and now enforce it to 1 day by default.

rf: https://www.rabbitmq.com/consumers.html#acknowledgement-timeout

To make the per queue argument feature work, update your RabbitMQ to version 3.12 at least.

If you want the old behavior where consumer timeout defaults to 30 minutes, you can either not upgrade your RabbitMQ or set `task.queue.consumer.timeout=30minutes` in `rabbitmq.properties`.

### RRT forwards now rewrite senders

Date: 09/10/2023
JIRA: https://issues.apache.org/jira/browse/JAMES-3944

In order to have valid SPF, DKIM configuration upon users forwards for mails sent from external address forwarded
to external addresses, James now rewrite the sender to the user owning the forward.

This behaviour can be disabled via the RecipientRewriteTable mailet configuration:

```
<mailet match="All" class="RecipientRewriteTable">
    <rewriteSenderUponForward>false</rewriteSenderUponForward>
<mailet/>>
```

Note that now `JMAPFiltering` mailet requires the `rrt-error` processor for handling forward loops. We thus recommend adding
this processor, that is part of the default configuration of your distribution.

### Change compaction strategy of blob_cache table
Date: 18/09/2023

To decrease the pressure on the Cassandra server.
The`blob_cache` table needs to use STCS instead of TWCS compaction strategy.

```
ALTER TABLE james_keyspace.blob_cache WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
```

### Set up TTL on the mailbox_change and email_change tables
Date: 15/09/2023

JIRA: https://issues.apache.org/jira/browse/JAMES-3940

Concerned products: Distributed James, Cassandra James Server

From now on, we set a default 60 days (configurable) Cassandra time-to-live on the `mailbox_change` and `email_change` tables 
to reduce considered outdated changes' storage usage.

If you want to keep the old behavior which is not using Cassandra time-to-live on those JMAP change tables, please set the 
following configurations in `cassandra.properties`:

```
email.change.ttl=0 second
mailbox.change.ttl=0 second
```

### Improve CassandraThreadIdGuessingAlgorithm
Date: 14/09/2023

JIRA: https://issues.apache.org/jira/browse/JAMES-3937

Concerned products: Distributed James, Cassandra James Server

Following the CassandraThreadIdGuessingAlgorithm improvement, `threadTable` and `threadLookupTable` Cassandra tables can be deleted.

Note that this will cause a discontinuity in thread allocation: 2 threads instead of one. This seems acceptable and preferable to a complex migration in our eyes.

### Drop Legacy Cassandra migrations

Date: 4/09/2023

We dropped the following Cassandra migrations and related tables after two years of support:

 - `FROM_V8_TO_V9`: `MessageV3Migration`
 - `FROM_V9_TO_V10`: `AclV2Migration`
 - `FROM_V10_TO_V11`: `MessageDenormalizationMigration`
 - `FROM_V11_TO_V12`: `AttachmentMessageIdMigration.class`

In order to run these migrations, users need to perform a rolling release to 3.8.0 and run these migration first.

### Mutualize quota table

Date: 24/08/2023
JIRA: https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3926

In order to upgrade to the latest Quota data model, one need to enable quota compatibility mode in `cassandra.properties`:

```
quota.compatibility.mode=true
```

This is required for going from Cassandra schema version 12 -> 15.

then run the migration: https://james.apache.org/server/manage-webadmin.html#Cassandra_Schema_upgrades

### Jmap Uploads

Date: 24/08/2023

`uploads` table can be deleted. Note that pending uploads will be lost during the migration, they are not used anymore.

### Imap Packages

Date: 27/06/2023

`imapserver.xml` needs to be modified to relocate `imapPackages` from imapServers tag into `imapServer` tags.

## 3.8.2 version

No specific operation to conduct from a 3.8.1 version.

## 3.8.1 version

No specific operation to conduct from a 3.8.0 version.

## 3.8.0 version

Changes to apply between 3.7.x and 3.8.0 are reported here.

Change list:

- [Adding authorized_users column to user table](#adding-authorized_users-column-to-user-table)
- [Migration to Cassandra driver 4](#migration-to-cassandra-driver-4)
- [Migration to OpenSearch](#migration-to-opensearch)
- [Deleted message vault is now deactivated by default](#deleted-message-vault-is-now-deactivated-by-default)
- [SortOrder addition in Identity](#sortorder-addition-in-identity)
- [Blob Store AES upgraded to PBKDF2WithHmacSHA512](#blob-store-aes-upgraded-to-pbkdf2withhmacsha512)
- [Adding saveDate column to messageIdTable and imapUidTable](#adding-savedate-column-to-messageidtable-and-imapuidtable)
- [Adding the saveDate to the OpenSearch index](#adding-the-savedate-to-the-opensearch-index)
- [Adding delegatedUser column to user_table](#adding-delegated_users-column-to-user-table)
- [DeletedMessageVaultHook should not be used on Cassandra based products](#deleted-message-vault-is-now-deactivated-by-default)

### JMAP filters event sourcing increments and snapshots

Date: 19/04/2023

JIRA: https://issues.apache.org/jira/browse/JAMES-3777

Concerned products: Distributed James, Cassandra James Server

Cassandra Event store table needs to be modified in order to support snapshots:

```
ALTER TABLE james_keyspace.eventStore ADD snapshot int static;
```

We recommend disabling increments during rolling updates (as older nodes in the cluster won't support them).
This can be done with the following system property:

```
-Djames.jmap.filters.eventsource.increments.enabled=false
```

### DeletedMessageVaultHook should not be used on Cassandra based products

Date: 10/02/2023

JIRA: https://issues.apache.org/jira/browse/JAMES-3882

Concerned products: Distributed James, Cassandra James Server

When the deleted message vault is enabled in Cassandra products, now the deleted messages are directly copied
into the vault, asynchronously upon deletions. Configuring DeletedMessageVaultHook is thus no longer needed.

### Adding delegated_users column to user table

Date 05/01/2023

JIRA: https://issues.apache.org/jira/browse/JAMES-3756

Concerned product: Distributed James, Cassandra James Server

Add `delegated_users` column to `user` tables in order to store delegated users that user has access to.

In order to add this `delegated_users` column you need to run the following CQL commands:
```
ALTER TABLE james_keyspace.user ADD delegated_users set<text>;
```

### Adding the saveDate to the OpenSearch index

Date 05/12/2022

JIRA: https://issues.apache.org/jira/browse/JAMES-3754

Concerned product: Distributed James, Cassandra James

Add `saveDate` to James mailbox index to enable saveDate searching as part of IMAP SAVEDATE extension.

We already have this field as part of newly created mappings, but we need to explicitly add this field to existing indices by doing:
```
curl -X PUT \
  http://ip:port/mailbox_v1/_mapping \
  -H 'Content-Type: application/json' \
  -d "{
	\"properties\": {
		\"saveDate\": {
			\"type\": \"date\",
			\"format\": \"uuuu-MM-dd'T'HH:mm:ssX||uuuu-MM-dd'T'HH:mm:ssXXX||uuuu-MM-dd'T'HH:mm:ssXXXXX\"
		}
	}
}"
```

### Adding saveDate column to messageIdTable and imapUidTable

Date 01/12/2022

JIRA: https://issues.apache.org/jira/browse/JAMES-3754

Concerned product: Distributed James, Cassandra James Server

Add `save_date` column to `messageIdTable` and `imapUidTable` tables in order to store saveDate data as part of IMAP SAVEDATE extension.

In order to add this `messageIdTable` column you need to run the following CQL commands:
```
ALTER TABLE james_keyspace.messageIdTable ADD save_date timestamp;
ALTER TABLE james_keyspace.imapUidTable ADD save_date timestamp;
```

### Blob Store AES upgraded to PBKDF2WithHmacSHA512

Date: 06/10/2022

PBKDF2WithHmacSHA1 was reported being a weak cryptographic set up. 

As such we decided to rely on PBKDF2WithHmacSHA512 by default.

Users having data encrypted with PBKDF2WithHmacSHA1 can configure it as a private key algorithm to
access their data, in `blobstore.properties`:

```
encryption.aes.private.key.algorithm=PBKDF2WithHmacSHA1
```

We do not have a migration strategy for PBKDF2WithHmacSHA1 users to migrate to PBKDF2WithHmacSHA512.

### SortOrder addition in Identity

Date: 06/10/2022

Concerned products: Distributed James, Cassandra James

JIRA: https://issues.apache.org/jira/browse/JAMES-3831

We added a `sortOrder` field to the identity object. Modification to the underlying Cassandra table is needed:

```
cqlsh:apache_james> ALTER TABLE custom_identity ADD sort_order int  ;
```

### Deleted message vault is now deactivated by default

Date: 13/09/2022

Concerned products: Distributed James, Cassandra James

In order to limit Cassandra table count, amongst others, the deleted message vault is now 
disabled by default.

Users relying on it needs to enable it within `deletedMessageVault.properties` with the following 
property:

```
enabled=true
```

### Removal of cassandra.properties chunk.size.message.read

Date: 23/08/2022

Concerned products: Distributed James, Cassandra James

JIRA: https://issues.apache.org/jira/browse/JAMES-3809

This setting was duplicated and do not account for the fetch type. Removed to avoid confusion.

Use `batchsize.properties` configuration file instead.

### Migration to OpenSearch

Date: 01/08/2022

Concerned products: Distributed James, Cassandra James

ElasticSearch v7 being EOL and upper versions being moved to a non-OSI compliant license (java clients included),
it's been decided to migrate to OpenSearch that is a fork of ElasticSearch 7.10 and using Apache v2 license.

You should migrate your ElasticSearch servers from version 7 to OpenSearch 2.1 before upgrading James.

Users can either perform a migration of their data from version 7 to OpenSearch (documentation
[here](https://opensearch.org/docs/latest/upgrade-to/upgrade-to/)) or might start from an empty
Opensearch v2.1 cluster and [reindex data](https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#reindexing-all-mails).

Keep in mind as well that the group listeners related to ElasticSearch changed names.
You would need to manually delete the bindings from the event queues in RabbitMQ before restarting James.

Configuration file was relocated to `opensearch.properties` and properties are now prefixed by `opensearch.` rather than
`elasticsearch.`.

Furthermore, implementation name that could be specified explicitly in `search.properties` has changed from `elasticsearch`
to `opensearch`.

### Migration to Cassandra driver 4

Date: 07/06/2022

Concerned product: Distributed James, Cassandra James Server

Cassandra driver had been updated to its version 4, empowering better reactive chains,
more frequent releases, more configuration options amongst other.

Users need to be aware that Cassandra driver 4 no longer allow in depth programmatic
configuration, and instead relies on its own configuration mechanisms.

Especially, SSL, consistency levels, timeouts, slow logs, metrics are now only configurable
through the mean of Cassandra driver native configuration. We dropped related options from the `cassandra.properties`
file.

You can find here an [example](https://github.com/apache/james-project/blob/master/server/apps/distributed-app/sample-configuration/cassandra-driver.conf) of what `conf/cassandra-driver-conf` looks like, with sane defaults.


Formal driver documentation: https://docs.datastax.com/en/developer/java-driver/4.13/manual/core/configuration/

### Adding authorized_users column to user table

Date 26/05/2022

JIRA: https://issues.apache.org/jira/browse/JAMES-3756

Concerned product: Distributed James, Cassandra James Server

Add `authorized_users` column to `user` table in order to store delegated users.

In order to add this `authorized_users` column you need to run the following CQL command:
```
ALTER TABLE james_keyspace.user ADD authorized_users set<text>;
```

## 3.7.6 version

No specific operation to conduct from a 3.7.5 version.

## 3.7.5 version

No specific operation to conduct from a 3.7.4 version.

## 3.7.4 version

No specific operation to conduct from a 3.7.3 version.

## 3.7.3 version

Changes to apply between 3.7.2 and 3.7.3 are reported here.

Change list:

- [TLS host name verification is now enabled by default](#tls-host-name-verification-is-now-enabled-by-default)

### TLS host name verification is now enabled by default

Date: 06/10/2022

JIRA: JAMES-3833

When establishing an SMTPS or StartTLS connection during remote mail delivery, James will now check the remote server's host/domain name against its certificate (RFC 2595). If they do not match, the connection fails as a temporary delivery error.

This prevents attackers from spoofing legitimate servers and intercepting mails. However, it may prevent James from connecting to servers that have strange certificates, no DNS entries, are reachable by IP address only, and similar edge cases.

Users requiring such connectivity may disable this check within `mailetcontainer.xml` at the RemoteDelivery mailet configuration:

```
<verifyServerIdentity>false</verifyServerIdentity>
```

## 3.7.2 version

No specific operation to conduct from a 3.7.1 version.

## 3.7.1 version

No specific operation to conduct from a 3.7.0 version.

## 3.7.0 version

Changes to apply between 3.6.2 and 3.7.0 are reported here.

Change list:

 - [Adding the threadId to the ElasticSearch index](#adding-the-threadid-to-the-elasticsearch-index)
 - [Rework message denormalization](#rework-message-denormalization)
 - [Adding threadId column to message metadata tables](#adding-threadid-column-to-message-metadata-tables)
 - [Removal of old tables for Cassandra mailRepository](#removal-of-old-tables-for-cassandra-mailrepository)
 - [Adding messageId metadata to the Cassandra attachments](#adding-messageid-metadata-to-the-cassandra-attachments)
 - [Changes to the enqueuedMails DAO](#changes-to-the-enqueuedmails-dao)
 - [Restructure maximum quotas definition](#restructure-maximum-quotas-definition)
 - [Vacation Mailet moved](#vacation-mailet-moved)
 - [Support salted passwords](#support-salted-passwords)
 - [MailDir removal](#maildir-removal)
 - [Change defaults for JPA UsersRepository hash function](#change-defaults-for-jpa-usersrepository-hash-function)
 - [Restrict listening interface to loopback by default for webadmin](#restrict-listening-interface-to-loopback-by-default-for-webadmin)
 
### Change in behaviour for Bounce, NotifyPostmaster, NotifySender

Date: 03/02/2022

JIRA: https://issues.apache.org/jira/browse/JAMES-3712

These mailets uses the mail being processed to send a new email to a third party.

Prior to this change, the `prefix` property was prepended to the processed email and not to the one sent to the third party.
Ie: the bounce recipient will not have the prefix.

We changed that so that the `prefix` property is prepended to the new email and not to the processed one.
Ie: the bounce recipient will now have the prefix.

Users willing to retain the previous behaviour can do so by removing the `prefix` property from their mailet and add a 
`AddSubjectPrefix` after their mailet. Please note that most user can (and likely should) omit those 
modifications that are only mentioned here for the sake of completeness.

Here is an example of how to retain the strict behaviour applied prior this change for `Bounce`:

```
<!-- what was -->
<mailet matcher="All" class="Bounce>
  <prefix>[OUPS] </prefix>
</mailet>

<!-- can now be written -->
<mailet matcher="All" class="Bounce/>
<mailet matcher="All" class="AddSubjectPrefix>
  <subjectPrefix>[OUPS] </subjectPrefix>
</mailet>
```

### Restrict listening interface to loopback by default for webadmin

Date: 11/01/2022

JIRA: https://issues.apache.org/jira/browse/JAMES-3690

As a security safeguard, we now enforce the `host` configuration parameter defined in `webadmin.properties` when 
binding the listening interface for the webadmin server.

By default, it listens on the loopback interface.

The previous behaviour can be restored by using the following configuration:

```
host=0.0.0.0
```

Be aware that webadmin do not enforce authentication by default, and needs JWT protection to be safely exposed.

### SMTP/LMTP authentication configuration reworked

Date: 07/12/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3680

We reworked SMTP/LMTP configuration to improve security and better explicit the configuration effects.

We added a new setting within SMTP/LMTP configuration to avoid advertising authentication on unencrypted channels. 
For security reasons, it is enabled by default, which is a breaking change. To disable it:

```xml
    <smtpserver enabled="true">
        <jmxName>smtpserver-global</jmxName>
        <bind>0.0.0.0:25</bind>
        <!--- ... -->
        <auth>
            <announce>never</announce>
            <requireSSL>false</requireSSL>
        </auth>
        <!--- ... -->
    </smtpserver>
```

Also, `requireAuth` setting, which was misleading, had been renamed to `auth.announce`. This change did not affect
backward compatibility and `requireAuth` is still read as a fallback.

 - `<requireAuth>false</requireAuth>` can now be specified with `<auth><announce>never</announce></auth>`
 - `<requireAuth>announce</requireAuth>` can now be specified with `<auth><announce>always</announce></auth>`
 - `<requireAuth>true</requireAuth>` can now be specified with `<auth><announce>forUnauthorizedAddresses</announce></auth>`

Finally, an implicit wildcard value `0.0.0.0/0.0.0.0` was specified as authorizedAddresses (for James 2.3 backward 
compatibility) when `<requireAuth>false</requireAuth>` or (equivalent) `<auth><announce>never</announce></auth>`. 
This behaviour could result in a user believing she disabled authentication, but would ultimately result in an 
open relay, which is unsecure, and why we no longer apply this implicit value. Users relying on James as an open
relay will need to specify `authorizedAddresses` explicitly:

```xml
    <smtpserver enabled="true">
        <jmxName>smtpserver-global</jmxName>
        <bind>0.0.0.0:25</bind>
        <!--- ... -->
        <auth>
            <announce>never</announce>
        </auth>
        <authorizedAddresses>0.0.0.0/0.0.0.0</authorizedAddresses>
        <!--- ... -->
    </smtpserver>
```

### Vacation Mailet moved

Date: 29/11/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3675

The Vacation Mailet is now included in the standard mailet set, so you do not need to specify a package name anymore:

```
class="org.apache.james.jmap.mailet.VacationMailet"
=>
class="VacationMailet"
```

Please update your `mailetcontainer.xml` accordingly.

### Support salted passwords

Date: 24/11/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3674

With this update, the password algorithm may include an option to salt the password hash with the username, for increased security against rainbow table cracking. To use this feature, update your usersrepository.xml :
```
<algorithm>SHA-512/salted</algorithm>
```

Whenever users change their password, James will update the respective database entry with a salted hash.

Note that `plain` hashing mode is the default. If you were using `legacy` mode and want to keep it for some reason, you must specify this option explicitly:
```
<algorithm>SHA-512/legacy</algorithm>
```

### MailDir removal

Date 19/09/2021

Mailing list vote: https://www.mail-archive.com/server-dev@james.apache.org/msg71060.html

Concerned product: Spring configured with MailDir

The maildir implementation, deprecated in 3.6.1 release, following a non-interoperable James naming scheme, suffering
from numerous bugs and unmaintained for several years had been removed for the 3.7.0 release. 

We encourage maildir users to switch to the JPA implementation (relying on an embedded
database by default). Tools like [imapsync](https://imapsync.lamiral.info/) or 
[pop2imap](http://www.linux-france.org/prj/pop2imap/) can be used for this
migration. This [link](https://james.staged.apache.org/james-project/3.7.0/servers/distributed/operate/migrating.html) 
explains how to use IMAP Sync.

### Change defaults for JPA UsersRepository hash function

Date 27/08/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3637

We changed the default algorithm to hash user password with JPA
with SHA-512 instead of insecure SHA-1 / MD5.

As such, upgrades might result in inability for users to login, unless you
explicitly configure the unsecure algorithm you were using.

Changing the user password will upgrade them to the newly introduced hashing standards.

### Restructure maximum quotas definition

Date 06/08/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3630

Concerned product: Distributed James and Cassandra James server

We restructured maximum global quota in order to store them in a single Cassandra row. This enables to retrieve this 
data on a single primary key read, which is important for performance when receiving many emails over SMTP.

Please recreate the maximum quota after the upgrade. 

`defaultMaxQuota` table can be dropped.

### Changes to the enqueuedMails DAO

Date 06/08/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3629

Concerned product: Distributed James

In order to avoid uses of non-frozen Cassandra collection by the MailQueue component,
we introduced a new table, enqueuedMailsv4, not suffering from these pitfalls.

Pre-existing table, enqueuedMailsV3 is no longer read and can be dropped.

To avoid data loss (only noticeable when browsing the mailqueue email, no email
in flight in the mailqueue will be lost, they will just be non-browsable) do your
rolling upgrade with an empty mailqueue. 

To achieve an empty mailqueue, start a James server with SMTP and JMAP ports not exposed:
they will consume existing mails without creating new ones.
 
### Removal of old tables for Cassandra mailRepository

Date 13/08/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3631

Concerned products: Cassandra James, only for installation performed in 2018 or before (3.4.0).

To audit if you need to perform this upgrade run on a Cassandra server:

```
nodetool tablestats KEYSPACE mailrepositorycontent
```

A non-zero value in the key estimates means you need to perform a migration.

Migrating content of `mailrepositorycontent` table needs to be done prior a rolling upgrade. After the rolling upgrade 
the table can be removed.

In order to move content of the `mailrepositorycontent` table to `mailrepositorycontentv2` table for ``:

 - In `mailetcontainer.xml` define an orphan processor with just a ToRepository mailet for the processor you wishes to 
 migrate. Here:
 
```
        <processor state="migrate-address-error" enableJmx="true">
            <mailet match="All" class="ToRepository">
                <repositoryPath>cassandra://var/mail/address-error/</repositoryPath>
            </mailet>
        </processor>
```

 - (Restart James for the configuration changes to be applied...)

 - Then trigger a reprocessing from the mailRepository to the processor you created. This can be done via webamin:
 https://james.apache.org/server/manage-webadmin.html#Administrating_mail_repositories . Here:
 
```
curl -XPATCH http://ip:port/mailRepositories/var%2Fmail%2Faddress-error%2F/mails?action=reprocess&processor=migrate-address-error
```

 - Once the processing is finished you can clean up your configuration.

### Adding messageId metadata to the Cassandra attachments

Date 09/08/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3544

Concerned product: Distributed James, Cassandra James server

In order to simplify the query flow, and reduce the amount on primary key reads when working with attachments when using
JMAP, we did add the messageId property to the `attachmentV2` table.

You need to perform the corresponding schema changes prior an attempt for a rolling upgrade:

```
cqlsh:apache_james> ALTER TABLE attachmentV2 ADD message_id timeuuid ;
```

Newly stored attachment will then store their corresponding messageId alongside `attachmentV2` and will no longer need
to read `messageIdTable`.

We furthermore propose a migration task for adding `message_id` field to `attachmentV2` and thus 
ensure you leverage the benefits of this change:

See how to [upgrade to the latest schema version](https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#upgrading-to-the-latest-version) (V12).

### Adding the threadId to the ElasticSearch index

Date 22/07/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3516

Concerned product: Distributed James

Add threadId to James document mapping to enable thread query search.

We already have this field as part of newly created mappings, but we need to explicitly add this field to existing indices by doing:
```
curl -X PUT \
  http://ip:ESport/mailbox_v1/_mapping \
  -H 'Content-Type: application/json' \
  -d '{
    "properties": {
        "threadId": {
            "type": "keyword"
        }
    }
}'
```

### Rework message denormalization

Date: 02/05/2021
JIRA: https://issues.apache.org/jira/browse/JAMES-3576
Concerned products: Cassandra Guice servers (including Distributed server)

The table structure had been reworked in order to fasten message reads (metadata and headers) workloads for both IMAP
and JMAP protocols.

Information previously included solely in `messagev3` table (and thus needed to be read) have been added to both 
`messageIdTable` (serving IMAP reads) and `imapUidTable` (serving JMAP reads), allowing skipping reading messageV3
table when retrieving messages METADATA and HEADERS, effectively decreasing Cassandra query workload.

Before deploying the new version of James one need to previously adapt the table structure:

```
cqlsh:apache_james> ALTER TABLE messageIdTable ADD internalDate timestamp ;
cqlsh:apache_james> ALTER TABLE messageidtable ADD bodyStartOctet int  ;
cqlsh:apache_james> ALTER TABLE messageidtable ADD fullContentOctets bigint  ;
cqlsh:apache_james> ALTER TABLE messageidtable ADD headerContent text  ;

cqlsh:apache_james> ALTER TABLE imapUidTable ADD internalDate timestamp ;
cqlsh:apache_james> ALTER TABLE imapUidTable ADD bodyStartOctet int  ;
cqlsh:apache_james> ALTER TABLE imapUidTable ADD fullContentOctets bigint  ;
cqlsh:apache_james> ALTER TABLE imapUidTable ADD headerContent text  ;
```

Note that if for some message information is missing from `messageIdTable` or `imapUidTable` then `messagev3` table will
effectively be read, defaulting to previous behaviour.

We furthermore propose a migration task for adding missing information to `imapUidTable` and `messageIdTable` and thus 
ensure you leverage the benefits of this change:

See how to [upgrade to the latest schema version](https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#upgrading-to-the-latest-version) (V11).

Change list:

 - [Drop Cassandra schema version prior version 8](#drop-cassandra-schema-version-prior-version-8)
 - [Adopt UnboundID as a LDAP library](#adopt-unboundid-as-a-ldap-library)
 - [Review the architecture of the RabbitMQ event bus](#review-the-architecture-of-the-rabbitmq-event-bus)
 - [Adding threadId column to message metadata tables](#adding-threadid-column-to-message-metadata-tables)
 
### Adding threadId column to message metadata tables

Date 23/06/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3516

Concerned product: Distributed James

Add threadId column to messageIdTable and imapUidTable in order to get a message's threadId.

In order to add this threadId column we advise you to run the following commands:
```
ALTER TABLE james_keyspace.messageIdTable ADD threadId timeuuid;
ALTER TABLE james_keyspace.imapUidTable ADD threadId timeuuid;
```

### Review the architecture of the RabbitMQ event bus

Date 14/06/2021

JIRA: https://issues.apache.org/jira/projects/JAMES/issues/JAMES-3599

Impacted products: Distributed James server

We now group listeners execution whenever possible. This minimizes:
                                                    
 - The count of events to deserialize (one for all groups)
 - The count of ACKs to perform

Note that retries are still performed on a per-group basis.

One need, after a rolling upgrade to unbind group queues from the primary exchange. Group queues can be identified by
their `mailboxEvent-workQueue-` prefix, and the primary exchange is named `mailboxEvent-exchange`. These operations can
easily be performed via the rabbitMQ management web interface.
 
### Adopt UnboundID as a LDAP library

Date 09/06/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3594

The previous LDAP implementation was based on JNDI and was causing operational concerns by opening a connection
for each user authentication. These limitations were inherent to JNDI thus to mitigate those we decided to migrate to
a newer LDAP library: [UnboundID](https://ldap.com/unboundid-ldap-sdk-for-java/).

As part of this migration the following change took place:

 - `useConnectionPool` : Removed. UnboundId implementation relies on a pool by default.
 - `poolSize` : Added. Allow controlling the count of connection in the pool.
 - Retries had been removed. Invalid connections errors are retries. THe following parameters were removed:
 `maxRetries`, `retryStartInterval`, `retryMaxInterval`, `retryIntervalScale`. Those values will be ignored.
 
The "group restriction" feature should furthermore be considered experimental, its usage is discouraged.

### Drop Cassandra schema version prior version 8

Date 05/05/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3578

Concerned products: Distributed James, Cassandra-guice James server

In an effort to simplify the code base, we dropped support for Cassandra schema version prior version 8.

Installation running older schema version than version 5 needs to rely on releases 3.6.x to upgrade to schema version 8,
before upgrading to an eventual newer version.

## 3.6.2 version

No specific operation to conduct from a 3.6.0 version.

## 3.6.1 version

No specific operation to conduct from a 3.6.0 version.

## 3.6.0 version

Changes to apply between 3.5.0 and 3.6.0 are reported here.

Change list:

 - [Drop Cassandra schema version prior version 5](#drop-cassandra-schema-version-prior-version-5)
 - [mailqueue.size.metricsEnabled now defaults to false](#mailqueuesizemetricsenabled-now-defaults-to-false)
 - [LDAP users repository connection pool now defaults to false](#ldap-users-repository-connection-pool-now-defaults-to-false)
 - [Swift support has been dropped](#swift-support-has-been-dropped)
 - [Cassandra Schema update to V8](#cassandra-schema-update-to-v8)
 - [Cassandra Schema update to V9](#cassandra-schema-update-to-v9)
 - [Cassandra Schema update to V10](#cassandra-schema-update-to-v10)
 - [Recommended upgrade to Cassandra 3.11.10](#recommended-upgrade-to-cassandra-3-1110)
 - [JMS mail queue no longer relies on java serialization](#jms-mail-queue-no-longer-relies-on-java-serialization)
 - [CassandraUsersRepository hash algorithm changes](#cassandrausersrepository-hash-algorithm-changes)
 - [Recommended upgrade to Elasticsearch 7.10.2](#recommended-upgrade-to-elasticsearch-7102)

### CassandraUsersRepository hash algorithm changes

Date: 05/03/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3512

Concerned products: Cassandra Guice servers (without LDAP)

Previous computation was not flushing the base64 encoding stream resulting in the 2 last characters of the base64 
string being omitted, causing inter-operability issues when synchronizing password hash with external systems.

New digest algorithm closes the base64 encoding stream resulting in a breaking change forcing user to reset their 
password.

If you wish to avoid this breaking change you can add the `hashingMode` configuration option in `usersrepository.xml` 
configuration file, with the value `legacy`, which will ensure previous behaviour.

Example `usersrepository.xml`:

```
<usersrepository>
    <algorithm>SHA-512-legacy</algorithm>
    <hashingMode>legacy</hashingMode>
    <!-- ... -->
</usersrepository>
```

### Recommended upgrade to Elasticsearch 7.10.2

Date: 11/03/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-3492

Concerned products: Cassandra Guice server (with and without rabbitMQ)

James is no longer compatible with Elasticsearch 6.x. The new environment uses Elasticsearch 7.10.2 and users are recommended to upgrade to this
version as well.

This [link](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/rolling-upgrades.html) describes a possible update procedure:

Updating from Elasticsearch < 6.8 needs an extra step of shutting down the cluster and restarting it after each node is upgraded

- Test the upgrade in dev environment first
- Take a backup of the cluster
- On each node, perform a rolling update
- Disable shard allocation
- Stop non-essential indexing and perform a synced flush this node
- Shut down this node.
- Upgrade the node you shut down
- Upgrade any plugins
- Start the upgraded node
- Reenable shard allocation
- Wait for the node to recover
- Repeat for other nodes

### JMS mail queue no longer relies on java serialization

Date: 07/12/2020

JIRA: https://issues.apache.org/jira/browse/JAMES-2578

Concerned products: Spring server, JPA Guice servers, Cassandra Guice server (without rabbitMQ)

Java serialization is unsafe and leads to countless vulnerability issues. We removed the remaining usages as part of the
JMS and ActiveMQ mail queues.

In order to do so, we introduced a breaking change in the message format.

Upgrades should be done with an empty mail queue. To do so:
 - Switch off incoming SMTP + JMAP traffic (external load balancing or restarting with dis-activated SMTP service)
 - Flush your MailQueues
 - Await the mail processing to finish, monitoring the size of your mail queue
 - Upgrade, then restart SMTP + JMAP traffic

### Recommended upgrade to Cassandra 3.11.10

Date: 15/02/2021

JIRA: https://issues.apache.org/jira/browse/JAMES-2514

Concerned products: Cassandra Guice server (with and without rabbitMQ)

James is no longer tested against Cassandra 3.11.3 but instead against Cassandra 3.11.10. Users are recommended to upgrade to this
version as well.

This [link](https://www.instaclustr.com/support/documentation/cassandra/cassandra-cluster-operations/cassandra-version-upgrades/#) describes a possible update procedure:

 - Take a backup of the cluster
 - On each node, perform a rolling update
 - For each node flush the data (`nodetool flush`)
 - Stop C* on this nod and upgrade to target version (approx 5-10 minutes duration)
 - Start upgraded C* on this node.
 - Confirm the application behaves normally
 - Upgrade remaining nodes
 - No upgradesstables is required for this minor version
 - Perform final verifications.

### Cassandra Schema update to V10

Date 05/12/2020

JIRA: https://issues.apache.org/jira/browse/JAMES-3435

Concerned product: Distributed James

Handles Mailbox ACL transactionality with event-sourcing. We got read of SERIAL consistency upon reads thus unlocking a
major performance enhancement.

In order to benefit from this work, you need to [upgrade to the latest schema version](https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#upgrading-to-the-latest-version).

A James restart is advised after this migration in order to skip schema version reads.

### Cassandra Schema update to V9

Date 20/10/2020

JIRA: https://issues.apache.org/jira/browse/JAMES-3430

Concerned product: Distributed James

Adopt a more compact representation for message properties. This improves performance of the Cassandra mailbox.

In order to benefit from this work, you need to [upgrade to the latest schema version](https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#upgrading-to-the-latest-version).

### Cassandra Schema update to V8

Date 13/10/2020

JIRA: https://issues.apache.org/jira/browse/JAMES-3409

Concerned product: Distributed James

Add UID_VALIDITY to mailboxPath table in order not to mandate mailbox table reads. This improves performance of the Cassandra mailbox.

In order to benefit from this work, you need to [upgrade to the latest schema version](https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#upgrading-to-the-latest-version).

A James restart is advised after this migration in order to skip schema version reads.

### Swift support has been dropped

Date 14/09/2020

JIRA: https://issues.apache.org/jira/browse/JAMES-3028

Concerned product: Distributed James

OpenStack's Swift APIs support has been dropped, please use Cassandra or AWS S3 instead. Please note that modern Swift 
versions support S3 APIs out of the box, thus OpenStack Swift can still be used through the S3 API.

### LDAP users repository connection pool now defaults to false

Date 08/07/2020

JIRA: https://issues.apache.org/jira/browse/JAMES-3300

Concerned product: all products relying on LDAP users repository

`useConnectionPool` is now false by default. If you really want it, you have to explicitly put it to `true` in `usersrepository.xml`.
It is false by default because it can create too many connections on the LDAP server. If you have few users it can eventually still make sense.

### mailqueue.size.metricsEnabled now defaults to false

Date 03/06/2020

JIRA: https://issues.apache.org/jira/browse/JAMES-2760

Concerned product: Distributed James

`mailqueue.size.metricsEnabled` is now false by default. If you previously used it, please be aware that it can have
some important performance penalty, and set it explicitly to `true` if you still need it.

### Drop Cassandra schema version prior version 5

Date 06/04/2020

JIRA: https://issues.apache.org/jira/browse/JAMES-2997

Concerned product: Distributed James, Cassandra-guice James server

In an effort to simplify the code base, we dropped support for Cassandra schema version prior version 5.

Installation running older schema version than version 5 needs to rely on release 3.5.0 to upgrade to schema version 7,
before upgrading to an eventual newer version.


## 3.5.0 version

Changes to apply between 3.4.0 and 3.5.0 are reported here.

Change list:

 - [LinShare blob export mechanism should rely on delegation](#LinShare blob export mechanism should rely on delegation)
 - [RabbitMQ minimal version](#rabbitmq-minimal-version)
 - [Enforce usernames to be lower cased](#enforce-usernames-to-be-lower-cased)
 - [Cassandra keyspace creation configuration](#cassandra-keyspace-creation-configuration)
 - [UsersFileRepository](#usersfilerepository)
 - [ElasticSearch performance enhancements](#elasticsearch-performance-enhancements)
 - [JAMES-2703 Post 3.4.0 release removals](#james-2703-post-340-release-removals)
 - [Health checks routes return code changes](#health-checks-routes-return-code-changes)
 - [User mailboxes reIndexing endpoint change](#user-mailboxes-reindexing-endpoint-change)
 - [Hybrid blobStore replaces Union blobStore](#hybrid-blobstore-replaces-union-blobstore)
 - [New forbidden set of characters in Usernames local part](#new-forbidden-set-of-characters-in-usernames-local-part)
 - [UidValidity and maildir](#uid-validity-and-maildir)
 - [UidValidity and JPA or Cassandra](#uid-validity-and-jpa-or-cassandra)
 - [Differentiation between domain alias and domain mapping](#differentiation-between-domain-alias-and-domain-mapping)
 - [ProtocolSession storng typing](#protocolsession-storng-typing)
 - [Tune Cassandra time serie tables options](#tune-cassandra-time-serie-tables-options)
 - [Log4J2 Adoption](#log4j2-adoption)

### Log4J2 Adoption

Date 20/03/2020

SHA-1 7682112258dc2e0b7322bfcf9cb44c1d287af422

JIRA: https://issues.apache.org/jira/browse/JAMES-3122

Concerned product: Spring

As Log4J 1.x is not compatible with Java 9+ runtime, we adopted Log4J2 as a logging solution for the Spring product.

As a consequence, the log configuration file for Spring product needs to be updated. See an example 
[here](server/app/src/main/resources/log4j2.xml).

Also, the deprecated `LogEnabled` API will be removed. We recommend extension developers to obtain their
Logger instance using the SLF4J `LoggerFactory` class instead.

### Tune Cassandra time serie tables options

Date 18/03/2020

SHA-1 bd6074cb689517061ecef9ddca3ab7a88d133cda

JIRA: https://issues.apache.org/jira/browse/JAMES-3121

Impacted product: Guice distributed James

Our usage of Cassandra for the time series have been improved by fine-tuning the compaction strategy and
the read repair option.

#### Upgrade procedure

In order to unlock these improvements we advise you running the following commands on existing instalations

```
ALTER TABLE james_keyspace.deletedMailsV2 WITH compaction = { 'class' :  'TimeWindowCompactionStrategy'  };
ALTER TABLE james_keyspace.enqueuedMailsV3 WITH  compaction = {'compaction_window_size': '1',
                                                               'compaction_window_unit': 'HOURS',
                                                               'class': 'org.apache.cassandra.db.compaction.TimeWindowCompaction' };
ALTER TABLE james_keyspace.applicableFlag WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.attachmentMessageId WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.attachmentV2 WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.firstUnseen WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.mailboxCounters WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.mailboxRecents WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.messageCounter WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.messageDeleted WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.modseq WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.imapUidTable WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.messageIdTable WITH compaction = { 'class' :  'SizeTieredCompactionStrategy'  };
ALTER TABLE james_keyspace.eventStore WITH compaction = { 'class' :  'LeveledCompactionStrategy'  };
ALTER TABLE james_keyspace.attachmentOwners WITH bloom_filter_fp_chance = 0.01;
ALTER TABLE james_keyspace.deletedMailsV2 WITH bloom_filter_fp_chance = 0.01;
ALTER TABLE james_keyspace.firstUnseen WITH bloom_filter_fp_chance = 0.01;
ALTER TABLE james_keyspace.mailboxCounters WITH bloom_filter_fp_chance = 0.01;
ALTER TABLE james_keyspace.messageCounter WITH bloom_filter_fp_chance = 0.01;
ALTER TABLE james_keyspace.deletedMailsV2 WITH read_repair_chance = 0.0;
ALTER TABLE james_keyspace.enqueuedMailsV3 WITH read_repair_chance = 0.0;
```

### ProtocolSession strong typing

Date 19/03/2020

SHA-1 58b1d879ab

JIRA: https://issues.apache.org/jira/browse/JAMES-3119

`ProtocolSession` have been reworked in order to increase type strength
and reduce errors.

Now `setAttachment` and `getAttachment` are expecting an `AttachmentKey` as key
and return an `Optional` instead of a `null`-able value.

Moreover `setAttachment` do not allow `null` values and `removeAttachment`
should be use now to remove elements.

### Differentiation between domain alias and domain mapping

Date 10/03/2020

SHA-1 81139cd94211358ca6a31b75b56c11b20e894ee4

JIRA: https://issues.apache.org/jira/browse/JAMES-3112

Concerned products: Guice products

We added strong typing for Domain aliases (exposed by this 
[endpoint](https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#get-the-list-of-aliases-for-a-domain))
to distinguish them from domain mappings (exposed by this 
[endpoint](https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#creating-address-domain-aliases)).

Read [this page](https://james.apache.org/server/config-recipientrewritetable.html) to understand the difference between 
Domain Alias and Domain Mapping.

As a consequence, existing values returned by the domain alias endpoint (before this fix this is domain mapping) will be 
considered as domain mappings and might need to be deleted and re-created.
 
### UidValidity and JPA or Cassandra

Date 26/02/2020

SHA-1 930fd38717b67f753b96f8b0595e80af7903298a

JIRA: https://issues.apache.org/jira/browse/JAMES-3074

Concerned products: JPA mailbox backend, Cassandra mailbox backend

Non-Maildir backends could generate '0' uid validity with a low probability, which is invalid regarding RFC-3501. 
We changed the generation mechanism to use valid UidValidity for newly created mailboxes. Regarding persisted mailboxes,
we regenerate invalid UidValidity upon reads.

While this sanitizing is transparent to the end user and the admin, it might lead to rare IMAP client full mailbox
 resynchronisation. (one chance out of two billions).

### UidValidity and maildir

Date 26/02/2020

SHA-1 837299290848cdc11a90c2e73c4af549a0f1106f

JIRA: https://issues.apache.org/jira/browse/JAMES-3074

Concerned products: Spring with maildir backend

Maildir generated too big values to conform to RFC-3501. We changed the generation mechanism to use valid UidValidity 
for newly created mailboxes. Regarding persisted mailboxes, we regenerate invalid UidValidity upon reads.

While this sanitizing is transparent to the end user and the admin, it might lead to IMAP client full resynchronisation.
 
### New forbidden set of characters in Usernames local part

Date 04/02/2020

SHA-1 4ebddfc107f7f47c29ff1d25dd045f314ff28672

JIRA: https://issues.apache.org/jira/browse/JAMES-2950

Even if this set of characters should be allowed for the local part of a Username regarding some context, as defined by the [RFC3696#section-3](https://tools.ietf.org/html/rfc3696#section-3), we decided that for reducing code complexity and more safety to not allow them anymore when creating new Usernames.

However, the read of Usernames already existing with some of those characters is still allowed, to not introduce any breaking change.

### LinShare blob export mechanism should rely on delegation
Date 12/02/2020

SHA-1 2422d3ef92a6c74e9da233039613d38fd833e632

Concerned products: Guice server, experimental LinShare blob export feature.

JIRA: https://issues.apache.org/jira/browse/JAMES-3040

Blob Export Configuration changed:

File configuration need to be adjusted: `blob.properties`

-You need to add new mandatory properties when `blob.export.implementation` property is set to `linshare`:
```
blob.export.linshare.technical.account.uuid
blob.export.linshare.technical.account.password
```

-The legacy property `blob.export.linshare.token` will not be used anymore, so you can remove it.
### Hybrid blobStore replaces Union blobStore

Date 6/01/2020

SHA-1 b44bc7d93901773ad442a341cb1ba2d94848c449

Concerned products: Guice distributed James server

Union blobStore, allowing to store older blobs within Cassandra while storing new blobs into object storage, had been removed.

Hybrid blobStore had been replacing it, allowing to store blobs either in a low cost blobStore or in a high performance blobStore, allowing thus some performance 
improvement for small, often read blobs while big or unfrequently read blobs keeps being stored cheaply.

Users relying on the Union blobStore will need to adopt Hybrid blobStore. Please adjust "blob.properties" accordingly:

```
implementation=hybrid
```

### User mailboxes reIndexing endpoint change
 
Date 16/12/2019

SHA-1 d68a64d96ba8312ecbd34761e82c9c2c32348290

Concerned products: Guice products

In an effort to have a resource oriented webadmin REST API we decided to relocate the endpoint for user mailbox reIndexing.

`curl -XPOST /mailboxes?task=reIndex&user=bob@apache.org`

Had been moved to:

`curl -XPOST /users/bob@apache.org/mailboxes?task=reIndex`

#### Health checks routes return code changes

Date 10/12/2019

SHA-1 cdbc0ee65f

Concerned products: Guice products

Health check return codes had been changed:
 - Degraded James server will now answer 200 instead of 500. JSON payload needs to be read in order to act upon a degraded server.
 - Unhealthy James server will now answer 503 instead of 500.
 
Depending on specific deployment usage of health-checks, management scripts might need to be updated.

#### RabbitMQ minimal version

Date 26/11/2019

SHA-1 0bf4e8384e

Concerned products: Guice distributed James (rabbitMQ)

The distributed James project (relying on Guice, Cassandra, ElasticSearch, RabbitMQ and optionally Swift) benefits from a new distributed task mananger.

In order to enforce task sequential processing at the cluster level, we rely on a single active consumer, which is a feature introduced in RabbitMQ 3.8.

Users of distributed James product thus need to upgrade their RabbitMQ server to be at least version 3.8.1.

#### Upgrade procedure

Ensure [no task is running or scheduled](http://james.apache.org/server/manage-webadmin.html#Listing_tasks).

 - Stop James nodes
 - drop `taskManagerWorkQueue` queue in RabbitMQ
 - Start new James nodes

#### Enforce usernames to be lower cased

Date 21/11/2019

SHA-1 9e976d3f49

JIRA: https://issues.apache.org/jira/browse/JAMES-2949

Many users recently complained about mails non received when sending to upper-cased local recipients. We decided to simplify the handling of case for local recipients and users by always storing them lower cased.
Now all the users repositories are storing user in lower case to ensure that. If you previously used to store users in a case-sensitive way (which is very unlikely as it is broking delivery), you could need to update your user database to lower case all your users.

#### Cassandra keyspace creation configuration

Date 15/11/2019

SHA-1 bcf4d36500

In order to allow the usage of cassandra credentials limited to a keyspace, the default behaviour of James is now to NOT
create the keyspace during start-up.

The automatic creation of the cassandra keyspace by James could be enabled by setting
 `cassandra.keyspace.create=true` in the `cassandra.properties` configuration file.
 
#### UsersFileRepository

Date 08/11/2019

SHA-1 0f8ee6ce2a

This specific user store was deprecated for years. It relied on Java serialization to store directly core.User object.

If you are still using it, you should instead use an [other user repository](https://james.apache.org/server/config-users.html).

If you need to export data from it, please [contact us](http://james.apache.org/#second), it could be possible to write a little extraction tool.

### ElasticSearch performance enhancements

Date 10/10/2019

SHA-1 0d72783ff4

JIRAS:
 - https://issues.apache.org/jira/browse/JAMES-2917
 - https://issues.apache.org/jira/browse/JAMES-2078
 - https://issues.apache.org/jira/browse/JAMES-2079
 - https://issues.apache.org/jira/browse/JAMES-2910

Concerned product: Guice product relying on ElasticSearch

We significantly improved our usage of ElasticSearch. Underlying changes include:

 - The use of routing to collocate emails of a same mailbox within a same shard. This enables search queries to avoid cluster
 level synchronisation, and thus enhance throughput, latencies and scalability.
 - Disabling dynamic mapping. We now represent headers as nested objects.
 - Removing some not needed fields from the mapping
 - No longer index raw HTML. This was possible under some configuration combination, and caused the data stored in elasticSearch
 to be significantly larger than required.

The downside of these changes is that a reindex is needed, implying a downtime on search:
 - Delete the indexes used by James
 - Start James in order to create the missing indexes
 - Trigger a [Full ReIndexing](https://james.apache.org/server/manage-webadmin.html#ReIndexing_all_mails), which can take
  time to complete.
 
#### JAMES-2703 Post 3.4.0 release removals

Date: 25/09/2019

SHA-1: f721747edf8deb50406a5a44f6476507a03e2543

JIRA: https://issues.apache.org/jira/browse/JAMES-2703

Concerned products: Spring, default bundled mailets

- Classes marked as deprecated whose removal was planned after 3.4.0 release (See JAMES-2703) had been removed. 

This includes:
  - SieveDefaultRepository. Please use SieveFileRepository instead.
  - JDBCRecipientRewriteTable, XMLRecipientRewriteTable, UsersRepositoryAliasingForwarding, JDBCAlias mailets. Please use RecipientRewriteTable mailet instead.
  - JDBCRecipientRewriteTable implementation. Please use JPARecipientRewriteTable instead.
  - JamesUsersJdbcRepository, DefaultUsersJdbcRepository. Please use JpaUsersRepository instead.
  - MailboxQuotaFixed matcher. Please use IsOverQuota instead.

## 3.4.0 version

Changes to apply between 3.3.x and 3.4.x will be reported here.

Change list:

 - [Upgrade to ElasticSearch 6.3](#upgrade-to-elasticsearch-63)
 - [Enqueuing several times a mail with the same name](#enqueuing-several-times-a-mail-with-the-same-name)
 - [RabbitMQ Mail Queue with multiple specific headers for a single recipient](#rabbitmq-mail-queue-with-multiple-specific-headers-for-a-single-recipient)

#### RabbitMQ Mail Queue with multiple specific headers for a single recipient

Date: 31/07/2019

SHA-1: f19642aef4a67cd6675f66278792ba4aa85d6d6e

JIRA: https://issues.apache.org/jira/browse/JAMES-2850

Concerned products: (experimental) RabbitMQ MailQueue

RabbitMQ MailQueue projection in Cassandra relies on a map allowing a single specific header per recipient header. This
limitation causes rejection of emails with multiple per-recipient headers (which happens when forwarding to a remote server
a mail checked against SpamAssassin).

In order to fix this issue, the structure of the underlying table was updated. A new table is created upon the first startup
following the upgrade, and the old one is ignored.

Impact: the mails enqueued before the update can not be browsed nor removed from the queue after the update.

Recommendation: Conduct this update with an empty mail queue.

To do so:

 - Given a distributed Guice James server
 - Stop incoming traffic
 - Monitor the mailQueues and wait them to be empty (spool & outgoing)
 - Once empty, upgrade James server and re-enable incoming traffic.

#### Enqueuing several times a mail with the same name

Date: XX/06/2019

SHA-1: XXXXX

JIRA: https://issues.apache.org/jira/browse/JAMES-2794

Concerned products: (experimental) RabbitMQ MailQueue

RabbitMQ mail queue combines RabbitMQ with projections in Cassandra to offer advanced management capabilities expected from
a mail queue (browse, delete, size, clear). In these projections, the mails are identified by their name. Thus, enqueuing a
mail that had already been processed will lead the given email to be considered already deleted, and it will be discarded 
and lost.

This is an issue, as several other components build features around submitting a mail several time with the name. 

For instance:

 - MailRepository reprocessing
 - RemoteDelivery bouncing under some configurations
 - RecipientRewriteTable rewriting to a remote server

We thus changed the table structure of RabbitMQ mail queue projections to be built around an EnqueueId. This additional 
level of indirection allows several enqueues with the same name.

Upgrade to the newest James server needs to be performed with an empty MailQueue.

To do so:

 - Given a distributed Guice James server
 - Stop incoming traffic
 - Monitor the mailQueues and wait them to be empty (spool & outgoing)
 - Once empty, upgrade James server and re-enable incoming traffic.

#### Upgrade to ElasticSearch 6.3

Date: 27/05/2019

SHA-1: bbdf88e56d7a22fe92e1360ef563004f3bc0dd98

JIRA: https://issues.apache.org/jira/browse/JAMES-2766

Concerned products: (experimental) Cassandra-guice products.

In version 3.3.0 indexing for the Cassandra product was handled using ElasticSearch 2.2 released on the 31 March 2016. Some major upgrades had been included in recent ElasticSearch version.

Note that ElasticSearch APIs had been undergoing some major changes, making a smooth migration hard to provide. We proposed 2 migration strategies. A
simple one leading to major search inconsistencies in the process, and another one mitigating these inconsistencies (but getting rid of them).

#### Configuration changes

ElasticSearch 6 driver is relying on the high-level REST client and no more on the internal transport protocol.

Thus, you need to update your configuration files accordingly:

In `elasticsearch.properties` modify the `elasticsearch.port` properties to reference the HTTP port of your ElasticSearch 
nodes (**9200** by default instead of the previous default value of **9300**).

##### Simple strategy

Procedure:
 - From a running James 3.3.0 cluster connected to a running ElacticSearch 2.2 cluster
 - Start an empty ElasticSearch 6.3 cluster
 - Shutdown James 3.3.0 cluster and start a James 3.4.0 cluster connected to ElasticSearch 6.3
 - Search result will then be empty and thus innacurate
 - Thus trigger a [Full ReIndexing](https://james.apache.org/server/manage-webadmin.html#ReIndexing_all_mails) to restore search consistency.

Keep in mind that full reIndexing needs to process all users email and thus can be slow.

Obviously this approach trades search consistency against ease of migration.

If search consistency during the migration is important for you, consider the next approach

##### Strategy for minimizing search inconsistency during the migration

Procedure:
 - From a running James 3.3.0 cluster connected to a running ElacticSearch 2.2 cluster
 - Start an empty ElasticSearch 6.3 cluster
 - Start a James 3.4.0 cluster connected to ElasticSearch 6.3 cluster as well as the Cassandra source of trust database. Traffic should be directed to the James 3.3.0 cluster.
 - Trigger an offline [Full ReIndexing](https://james.apache.org/server/manage-webadmin.html#ReIndexing_all_mails) on the James 3.4.0 cluster
 - Once done, direct the traffic to the James 3.4.0 cluster, and dispose the James 3.3.0 cluster as well as the ElasticSearch 2.2 cluster
 - Search result will omit changes that took place during the switching process (starting from the reIndexing start)
 - Thus trigger a [Full ReIndexing](https://james.apache.org/server/manage-webadmin.html#ReIndexing_all_mails) to restore search consistency.

Keep in mind that full reIndexing needs to process all users email and thus can be slow.

## 3.3.0 version

Changes to apply between 3.2.0 and 3.3.0 had been reported here.

Change list:

 - [Changes to the MailboxListener API](#changes-to-the-mailboxlistener-api)
 - [Changes in WebAdmin reIndexing API](#changes-in-webadmin-reindexing-api)
 - [Rename KEY column in JAMES_MAILBOX_ANNOTATION table](#james-mailbox-annotation)
 - [Mailet API changes](#mailet-api-changes)

### Changes to the MailboxListener API

#### Persistent MailboxId for MailDir

Date: 30/11/2018

SHA-1: 7e32da51a29bee1c732b2b13708bb4b986140119

JIRA: https://issues.apache.org/jira/browse/MAILBOX-292

MailboxId are now persisted in a `james-mailboxId` file. This file is created on the fly, so no action is required for users relying on
the MailDir mailbox.

#### Registration by MailboxId

Date: 30/11/2018

SHA-1: d9bcebc7dd546bd5f11f3d9b496491e7c9042fe2

JIRA: https://issues.apache.org/jira/browse/MAILBOX-354

Only user written components performing MailboxListener registration will be affected.

The MailboxPath is mutable and thus can be changed upon mailbox rename. This leads to significantly complex code with possible inconsistency windows.

Using the mailboxId, which is immutable, solves these issues.

### Changes in WebAdmin reIndexing API

Date: 05/12/2018

SHA-1: 985b9a4a75bfa75c331cba6cbf835c043185dbdb

JIRA: https://issues.apache.org/jira/browse/JAMES-2555

We made this API introduced in James 3.2.0 a bit more REST friendly. If you developed tools using this API, you will need to update them.

For more details please refer to [the latest WebAdmin documentation](https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#ReIndexing).

### Rename KEY column in JAMES_MAILBOX_ANNOTATION table

Date: 19/12/2018

SHA-1: e25967664538be18ec29f47e73e661bdf29da41f

JIRA: https://issues.apache.org/jira/projects/MAILBOX/issues/MAILBOX-356

Required: Yes

Concerned products: all JPA related products

#### Upgrade procedure

Rename `KEY` column in `JAMES_MAILBOX_ANNOTATION` table. The syntax is:

##### In MySQL
```
ALTER TABLE JAMES_MAILBOX_ANNOTATION CHANGE KEY ANNOTATION_KEY varchar(200);
```

##### In MariaDB
```
ALTER TABLE JAMES_MAILBOX_ANNOTATION CHANGE COLUMN KEY ANNOTATION_KEY varchar(200);
```

_or the syntax corresponding to your database._

### Mailet API changes
In order to allow safe serialization and strong typing `org.apache.mailet.Mail` have changed.

These methods have been deprecated and replaced:

 * `getSender()` in favor of `getMaybeSender()`
 * `getAttribute(String)` in favor of `getAttribute(AttributeName)`
 * `setAttribute(String, Serializable)` in favor of `setAttribute(Attribute)`
 * `removeAttribute(String)` in favor of `removeAttribute(AttributeName)`
 * `getAttributeNames()` in favor of `attributeNames()` and `attributesMap()`

Some plain-string `AttributeName` have also been replaced:

  * `SMTP_AUTH_USER_ATTRIBUTE_NAME` in favor of `SMTP_AUTH_USER`
  * `MAILET_ERROR_ATTRIBUTE_NAME` in favor of `MAILET_ERROR`
  * `SENT_BY_MAILET` in favor of `SENT_BY_MAILET_ATTRIBUTE`'s name, it is recommended to directly set the `Attribute`.

## 3.2.0 version

Changes to apply between 3.1.0 and 3.2.0 had been reported here.

Changelist:

 - [JMAPFiltering mailet is required for JMAP capable servers](#jmapfiltering-mailet-is-required-for-jmap-capable-servers)
 - [Cassandra 3.11.3 upgrade](#cassandra-3113-upgrade)

### Noticeable changes in Mail API: Mail::getMaybeSender

Date: 31/10/2018

SHA-1: 485406252d82c2d23a4078c76b26d6fc8973bbd7

JIRA: https://issues.apache.org/jira/browse/JAMES-2557

Required: Yes

Concerned products: User developed extensions - mailet/matcher

As part of the SMTP protocol, a mail can be sent without sender. This was represented implicitly in James by a potentially null MailAddress
(`null` or `MailAddress.nullSender()`). This means that mailet/matcher implementers needs to be aware, and handle these cases. This implicit
handling makes nullSender hard to work with, and prooved to be error-prone as part of the 3.2.0 development process.

Hence, we propose an alternative API returning a `MaybeSender` object, requiring the caller to explicitly handle missing sender.

`Mail::getSender` had then been deprecated. We strongly encourage our users to rely on `Mail::getMaybeSender`.

Note: thanks to java-8 default API methods, this is not a breaking change.

### JMAPFiltering mailet is required for JMAP capable servers

Date: 30/08/2018

SHA-1: 9ba6a1dd270f99735c7f9d3d4b2adb5076583c10

JIRA: https://issues.apache.org/jira/browse/JAMES-2529

Required: Yes

Concerned products: Cassandra Guice products

This mailet allow users filtering rules to be applied for incoming emails.

#### Upgrade procedure

Add this line before the `LocalDelivery` mailet of your `transport` processor:

```
<mailet match="RecipientIsLocal" class="org.apache.james.jmap.mailet.filter.JMAPFiltering"/>
```

### Cassandra 3.11.3 upgrade

Date: 03/08/2018

SHA-1: de0fa8a3df69f50cbc0684dfb1b911ad497856d7

JIRA: https://issues.apache.org/jira/browse/JAMES-2514

Required: Yes

Concerned products: Cassandra Guice products

James Cassandra Guice now officially uses Cassandra 3.11.3 as a storage backend. After performing the upgrade, the team
did perform some breaking changes, detailed below. James Cassandra Guice products are no more tested against Cassandra 2.2.x. Thus, we strongly
advise our users to upgrade.

#### Changes not compatible with Cassandra 2.2.x

Replace in default compaction strategies "DateTieredCompactionStrategy" by "TimeWindowCompactionStrategy".

This means you can no more start James on top of an empty Cassandra 2.2.x cluster, but existing deployments should not be impacted.

#### Upgrade procedure

We will assume that Cassandra had been installed with a debian package. Upgrade procedure stays similar in other cases.


1. Update Cassandra dists in `/etc/apt/sources.list.d/cassandra.list` to match 311x repository

```
deb http://downloads.apache.org/cassandra/debian 311x main
```


2. Update Cassandra

```
$ apt-get update
$ apt-get install cassandra=3.11.3
```

3. Correct the configuration

Edit /etc/cassandra/cassandra.yaml and ensure to really specify the interface cassandra is listening on as seeds.

4. ReStart Cassandra

4.1. Drain data & stop

```
$ nodetool drain
$ nodetool stop
```

4.2. start Cassandra

5. Upgrade SSTable (live update, performance degradation to expect)

```
$ nodetool upgradesstables apache_james
```
