This guide aims to be an entry-point to the James documentation for user
managing a {server-name}.

It includes:

* Simple architecture explanations
* Propose some diagnostics for some common issues
* Present procedures that can be set up to address these issues

In order to not duplicate information, existing documentation will be
linked.

Please note that this product is under active development, should be
considered experimental and thus targets advanced users.

== Basic Monitoring

A toolbox is available to help an administrator diagnose issues:

* xref:{xref-base}/operate/logging.adoc[Structured logging into Kibana]
* xref:{xref-base}/operate/metrics.adoc[Metrics graphs into Grafana]
* xref:{xref-base}/operate/webadmin.adoc#_healthcheck[WebAdmin HealthChecks]

== Mail processing

Currently, an administrator can monitor mail processing failure through `ERROR` log
review. We also recommend watching in Kibana INFO logs using the
`org.apache.james.transport.mailets.ToProcessor` value as their `logger`. Metrics about
mail repository size, and the corresponding Grafana boards are yet to be contributed.

Furthermore, given the default mailet container configuration, we recommend monitoring
`{mailet-repository-path-prefix}://var/mail/error/` to be empty.

WebAdmin exposes all utilities for
xref:{xref-base}/operate/webadmin.adoc#_reprocessing_mails_from_a_mail_repository[reprocessing
all mails in a mail repository] or
xref:{xref-base}/operate/webadmin.adoc#_reprocessing_a_specific_mail_from_a_mail_repository[reprocessing
a single mail in a mail repository].

In order to prevent unbounded processing that could consume unbounded resources. We can provide a CRON with `limit` parameter.
Ex: 10 reprocessed per minute
Note that it only support the reprocessing all mails.

Also, one can decide to
xref:{xref-base}/operate/webadmin.adoc#_removing_all_mails_from_a_mail_repository[delete
all the mails of a mail repository] or
xref:{xref-base}/operate/webadmin.adoc#_removing_a_mail_from_a_mail_repository[delete
a single mail of a mail repository].

Performance of mail processing can be monitored via the
https://github.com/apache/james-project/blob/d2cf7c8e229d9ed30125871b3de5af3cb1553649/server/grafana-reporting/es-datasource/MAILET-1490071694187-dashboard.json[mailet
grafana board] and
https://github.com/apache/james-project/blob/d2cf7c8e229d9ed30125871b3de5af3cb1553649/server/grafana-reporting/es-datasource/MATCHER-1490071813409-dashboard.json[matcher
grafana board].

=== Recipient rewriting

Given the default configuration, errors (like loops) uopn recipient rewritting will lead
to emails being stored in `{mailet-repository-path-prefix}://var/mail/rrt-error/`.

We recommend monitoring the content of this mail repository to be empty.

If it is not empty, we recommend
verifying user mappings via xref:{xref-base}/operate/webadmin.adoc#_listing_user_mappings_[User Mappings webadmin API] then once identified break the loop by removing
some Recipient Rewrite Table entry via the
xref:{xref-base}/operate/webadmin.adoc#_removing_an_alias_of_an_user[Delete Alias],
xref:{xref-base}/operate/webadmin.adoc#_removing_a_group_member[Delete Group member],
xref:{xref-base}/operate/webadmin.adoc#_removing_a_destination_of_a_forward[Delete forward],
xref:{xref-base}/operate/webadmin.adoc#_remove_an_address_mapping[Delete Address mapping],
xref:{xref-base}/operate/webadmin.adoc#_removing_a_domain_mapping[Delete Domain mapping]
or xref:{xref-base}/operate/webadmin.adoc#_removing_a_regex_mapping[Delete Regex mapping]
APIs (as needed).

The `Mail.error` field can help diagnose the issue as well. Then once
the root cause has been addressed, the mail can be reprocessed.

== Mailbox Event Bus

It is possible for the administrator of James to define the mailbox
listeners he wants to use, by adding them in the
{sample-configuration-prefix-url}/listeners.xml[listeners.xml]
configuration file. It’s possible also to add your own custom mailbox
listeners. This enables to enhance capabilities of James as a Mail
Delivery Agent. You can get more information about those
    xref:{xref-base}/configure/listeners.adoc[here].

Currently, an administrator can monitor listeners failures through
`ERROR` log review. Metrics regarding mailbox listeners can be monitored
via
https://github.com/apache/james-project/blob/d2cf7c8e229d9ed30125871b3de5af3cb1553649/server/grafana-reporting/es-datasource/MailboxListeners-1528958667486-dashboard.json[mailbox_listeners
grafana board] and
https://github.com/apache/james-project/blob/d2cf7c8e229d9ed30125871b3de5af3cb1553649/server/grafana-reporting/es-datasource/MailboxListeners%20rate-1552903378376.json[mailbox_listeners_rate
grafana board].

Upon exceptions, a bounded number of retries are performed (with
exponential backoff delays). If after those retries the listener is
still failing to perform its operation, then the event will be stored in
the xref:{xref-base}/operate/webadmin.adoc#_event_dead_letter[Event Dead Letter]. This
API allows diagnosing issues, as well as redelivering the events.

To check that you have undelivered events in your system, you can first
run the associated with
xref:{xref-base}/operate/webadmin.adoc#_healthcheck[event dead letter health check] .
You can explore Event DeadLetter content through WebAdmin. For
this, xref:{xref-base}/operate/webadmin.adoc#_listing_mailbox_listener_groups[list mailbox listener groups]
you will get a list of groups back, allowing
you to check if those contain registered events in each by
xref:{xref-base}/operate/webadmin.adoc#_listing_failed_events[listing their failed events].

If you get failed events IDs back, you can as well
xref:{xref-base}/operate/webadmin.adoc#_getting_event_details[check their details].

An easy way to solve this is just to trigger then the
xref:{xref-base}/operate/webadmin.adoc#_redeliver_all_events[redeliver all events]
task. It will start reprocessing all the failed events registered in
event dead letters.

In order to prevent unbounded processing that could consume unbounded resources. We can provide a CRON with `limit` parameter.
Ex: 10 redelivery per minute

If for some other reason you don’t need to redeliver all events, you
have more fine-grained operations allowing you to
xref:{xref-base}/operate/webadmin.adoc#_redeliver_group_events[redeliver group events]
or even just
xref:{xref-base}/operate/webadmin.adoc#_redeliver_a_single_event[redeliver a single event].

== OpenSearch Indexing

A projection of messages is maintained in OpenSearch via a listener
plugged into the mailbox event bus in order to enable search features.

You can find more information about OpenSearch configuration
xref:{xref-base}/configure/opensearch.adoc[here].

=== Usual troubleshooting procedures

As explained in the link:#_mailbox_event_bus[Mailbox Event Bus] section,
processing those events can fail sometimes.

Currently, an administrator can monitor indexation failures through
`ERROR` log review. You can as well
xref:{xref-base}/operate/webadmin.adoc#_listing_failed_events[list failed events] by
looking with the group called
`org.apache.james.mailbox.opensearch.events.OpenSearchListeningMessageSearchIndex$OpenSearchListeningMessageSearchIndexGroup`.
A first on-the-fly solution could be to just
link:#_mailbox_event_bus[redeliver those group events with event dead letter].

If the event storage in dead-letters fails (for instance in the face of
{backend-name} storage exceptions), then you might need to use our WebAdmin
reIndexing tasks.

From there, you have multiple choices. You can
xref:{xref-base}/operate/webadmin.adoc#_reindexing_all_mails[reIndex all mails],
xref:{xref-base}/operate/webadmin.adoc#_reindexing_a_mailbox_mails[reIndex mails from a mailbox] or even just
xref:{xref-base}/operate/webadmin.adoc#_reindexing_a_single_mail_by_messageid[reIndex a single mail].

When checking the result of a reIndexing task, you might have failed
reprocessed mails. You can still use the task ID to
xref:{xref-base}/operate/webadmin.adoc#_fixing_previously_failed_reindexing[reprocess previously failed reIndexing mails].

=== On the fly OpenSearch Index setting update

Sometimes you might need to update index settings. Cases when an
administrator might want to update index settings include:

* Scaling out: increasing the shard count might be needed.
* Changing string analysers, for instance to target another language
* etc.

In order to achieve such a procedure, you need to:

* https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-create-index.html[Create
the new index] with the right settings and mapping
* James uses two aliases on the mailbox index: one for reading
(`mailboxReadAlias`) and one for writing (`mailboxWriteAlias`). First
https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-aliases.html[add
an alias] `mailboxWriteAlias` to that new index, so that now James
writes on the old and new indexes, while only keeping reading on the
first one
* Now trigger a
https://www.elastic.co/guide/en/elasticsearch/reference/7.10/docs-reindex.html[reindex]
from the old index to the new one (this actively relies on `_source`
field being present)
* When this is done, add the `mailboxReadAlias` alias to the new index
* Now that the migration to the new index is done, you can
https://www.elastic.co/guide/en/elasticsearch/reference/7.10/indices-delete-index.html[drop
the old index]
* You might want as well modify the James configuration file
{sample-configuration-prefix-url}/opensearch.properties[opensearch.properties]
by setting the parameter `opensearch.index.mailbox.name` to the name
of your new index. This is to avoid that James re-creates index upon
restart

_Note_: keep in mind that reindexing can be a very long operation
depending on the volume of mails you have stored.

== Mail Queue

=== Fine tune configuration for RabbitMQ

In order to adapt mail queue settings to the actual traffic load, an
administrator needs to perform fine configuration tunning as explained
in
https://github.com/apache/james-project/blob/master/src/site/xdoc/server/config-rabbitmq.xml[rabbitmq.properties].

Be aware that `MailQueue::getSize` is currently performing a browse and
thus is expensive. Size recurring metric reporting thus introduces
performance issues. As such, we advise setting
`mailqueue.size.metricsEnabled=false`.

=== Managing email queues

Managing an email queue is an easy task if you follow this procedure:

* First, xref:{xref-base}/operate/webadmin.adoc#_listing_mail_queues[List mail queues]
and xref:{xref-base}/operate/webadmin.adoc#_getting_a_mail_queue_details[get a mail queue details].
* And then
xref:{xref-base}/operate/webadmin.adoc#_listing_the_mails_of_a_mail_queue[List the mails of a mail queue].

In case, you need to clear an email queue because there are only spam or
trash emails in the email queue you have this procedure to follow:

* All mails from the given mail queue will be deleted with
xref:{xref-base}/operate/webadmin.adoc#_clearing_a_mail_queue[Clearing a mail queue].

== Deleted Message Vault

We recommend the administrator to
xref:#_cleaning_expired_deleted_messages[run it] in cron job to save
storage volume.

=== How to configure deleted messages vault

To setup James with Deleted Messages Vault, you need to follow those
steps:

* Enable Deleted Messages Vault by configuring Pre Deletion Hooks.
* Configuring the retention time for the Deleted Messages Vault.

==== Enable Deleted Messages Vault by configuring Pre Deletion Hooks

You need to configure this hook in
{sample-configuration-prefix-url}/listeners.xml[listeners.xml]
configuration file. More details about configuration & example can be
found at http://james.apache.org/server/config-listeners.html[Pre
Deletion Hook Configuration]

==== Configuring the retention time for the Deleted Messages Vault

In order to configure the retention time for the Deleted Messages Vault,
an administrator needs to perform fine configuration tunning as
explained in
{sample-configuration-prefix-url}/deletedMessageVault.properties[deletedMessageVault.properties].
Mails are not retained forever as you have to configure a retention
period (by `retentionPeriod`) before using it (with one-year retention
by default if not defined).

=== Restore deleted messages after deletion

After users deleted their mails and emptied the trash, the admin can use
xref:{xref-base}/operate/webadmin.adoc#_restore_deleted_messages[Restore Deleted Messages]
to restore all the deleted mails.

=== Cleaning expired deleted messages

You can delete all deleted messages older than the configured
`retentionPeriod` by using
xref:{xref-base}/operate/webadmin.adoc#_deleted_messages_vault[Purge Deleted Messages].
We recommend calling this API in CRON job on 1st day each
month.
