Details regarding email deduplication - common Email Server behaviour with how they can create duplicate emails, and subsequent behaviour in Enate
As part of day-to-day processing in Email systems and integration with other systems, it is possible for errors to result in duplicate emails. To combat this Enate provides built-in capabilities to deduplicate received emails. Let’s take a look at the details for this.
As a core principle, Enate will generate or update 1 Work Item per received Email after deduplication.
To guard against accidental creation of multiple work items caused by duplicate incoming emails, Enate checks for such duplicates and deduplicates them so that only one work item is created. This is how Enate is set up as standard, and is termed ‘Global Level deduplication’.
However in some business operations, it is possible that the same exact email will be explicitly forwarded into two different parts of the business, with the intention of creating two different work items in Enate. In order to allow this, Enate supports the following options:
Global Level Deduplication (default) – where emails deduplication logic is run globally on all emails, and any mails considered duplicate are always deduped.
Per-Connector Level Deduplication – where emails are deduped if received into the same individual Email Connector, but are not deduped if the two mails are sent into two separate Email Connectors in Enate.
In Per-Connector mode, Enate will only search for existing Emails that have been received to the same Email Connector as configured in Enate.
An email is considered a duplicate in Enate when byte-for-byte 100% of the content is identical. This includes not just the user-visible content such as Subject, Body, Senders, Recipient, Attachments, etc but also the underlying email information such as the email headers.
In a scenario where a sender includes multiple addresses in the recipient list (To and CC) which are picked up by Enate, the system would normally receive 2 (or more) copies of the email, and the Email routing outside of Enate would determine whether it is caught as a duplicate or not. However, the duplication taking place in the Email Server infrastructure cannot be guaranteed.
Regardless of Global vs per-Connector deduplication, 2 (or more) copies of the email must be generated by the Email Server infrastructure otherwise the core principle stated above results in just a single Work Item.
Whether multiple emails are created or not in this scenario is outside the control of Enate and should be investigated with your Email Server provider or Email administration team. In our experience factors which may affect this include (but are not limited to):
If the email is sent by internal or external senders
If the recipients are real mailboxes or aliases
If transport rules are used (Exchange/Office 365)
If 3rd party antispam/antivirus products are involved in email delivery.
Due to the variability in behaviour of external systems with regards to multiple emails and duplicates, we strongly advise business processes and email scenarios are tested thoroughly to ensure the desired behaviour in Enate is achieved.
When Enate receives and processes multiple copies of the same Email, but with different headers, then multiple Work Items will be created. However each Email is necessarily processed and is completely unaware of the others.
If the Emails are received into the same mailbox, then the same Enate routing will be applied resulting in all the created Work Items being in the same context (and Ticket Category for Tickets).
If the Emails are received into different mailboxes, then each will follow the configured routing within that mailbox - these may obviously be different, resulting in the created Work Items potentially being in different contexts (and Ticket Categories for Tickets).
This limitation is caused by Email standards outside of Enate’s control, as explained in the Example below.
When an email is received because it has been BCC’d to the email address of the mailbox, or one of its aliases, configured in Enate, the system may not be able to route the email. This is because the BCC (Blind Carbon Copy) details are intentionally hidden from all recipients as part of standard Email behaviour. The result of this is that Enate receives an Email that does not identify itself as being destined for any configured Email address.
In this scenario an automated rejection Email will be sent informing the sender that they emailed an unconfigured address.
Imagine an email sent to joe@enate.net, tom@enate.net and joe@enate.io and BCCd to kevin@enate.net. Here joe@enate.io is an alias for joe@enate.net.
If the email is sent by a colleague (an internal email) then it is likely, depending on Email Infrastructure, that Joe would receive 1 email, Tom would receive 1 email and Kevin would receive 1 email.
If the email is sent by an external contact (an external email) then it is likely, depending on Email Infrastructure, that Joe would receive 2 emails, Tom would receive 1 email and Kevin would receive 1 email. Joe would see 2 identical emails in his Inbox with no discernible differences; only when viewing the Email headers would differences be identified and even then it is unlikely either email could be attributed to joe@enate.net vs joe@enate.io.
In both examples Kevin would receive the email but wouldn’t really know why – he would not be able to identify himself as a recipient in the To or CC fields and the BCC field is not shown.
The impact of these 2 examples if received by Enate is that for the first example (sent from internal):
1 email would be received in the mailbox joe@enate.net which would follow the configured routing rules resulting in 1 Work Item.
1 email would be received in the mailbox tom@enate.net which would follow the configured routing rules resulting in 1 Work Item.
1 email would be received in the mailbox kevin@enate.net but routing rules on this Email Connector wouldn’t match any of the To or CC addresses, so a rejection Email would be sent.
In the second example (sent from external):
2 emails would be received in the mailbox joe@enate.net which would follow the configured routing rules resulting in 2 Work Item in the same Context/Ticket Category. This is because Enate looks for a Routing Rule by evaluating the combined To/CC list reading from left-to-right with To taking precedence over CC. There is nothing in the two emails to identify each one as being intended for a different address; in that regard they are identical.
1 email would be received in the mailbox tom@enate.net. Same as for Internal, behaviour would be as above would result in 1 work item.
1 email would be received in the mailbox kevin@enate.net. Same as for Internal, behaviour as above would result in a rejection Email and no work item.
