We provisioned a Microsoft 365 tenant, populated it with a 10,000-mail synthetic corpus and 1,700 attachments across 8 file-type subtypes, then benchmarked NinjaOne SaaS Archiver, Barracuda Cloud Archiving Service, Acronis Cyber Protect Cloud Email Archiving, and MailPiler on the same tenant against 10 dimensions covering ingestion, search, attachment recall, export, immutability, legal hold, audit, encryption, retention, and vendor access posture.
Email archiving benchmark results
- Overall Score: weighted sum of 10 scoring categories on a 0-100 scale. See email archiving benchmark category scores for the weights.
- Subtypes: number of the 8 attachment file-types (PDF text-layer, PDF scanned, DOCX, XLSX, PPTX, PNG, CSV, ZIP) where the “Internal Document” marker phrase was indexable in the vendor’s search.
- Export: bulk export options available from the customer UI in a single step.
Key findings by product
- NinjaOne: Wins overall with a 1-point lead driven by perfect scores on immutability, legal hold, audit log, and export integrity, plus the only vendor-shipped SHA256 checksum on its bulk export bundle.
- Barracuda: 3 Microsoft consent permissions, no admin user creation in the tenant, and customer-managed journal rule artifacts.
- Acronis (encryption): The only product offering customer-set encryption (BYOK) in the customer UI. The customer enters a password during plan creation, and that password becomes the per-archive encryption key.
- Acronis (export): Also, the only product without bulk EML or PST export. Bulk actions on search results are limited to Send as email, Recover-to-mailbox (re-injection via Microsoft 365 transport), or Download attachments, capped at 10 emails per batch.
- Piler: Captures 54% of the 10,000-mail corpus because its inbox-forward architecture cannot see mail that Microsoft 365 silently quarantined before delivery; commercial vendors capture at submit time and reach 100%. Covers 4 of 8 attachment subtypes (PDF text-layer, DOCX, PPTX, and ZIP container unwrap) and misses scanned PDF OCR, XLSX, PNG, and CSV plaintext indexing.
- Test environment: We deployed each product on the same Microsoft 365 tenant bencharchive.onmicrosoft.com, one user mailbox (user1) as the universal recipient, and 25 shared sender mailboxes cycled as senders during corpus generation.
- Scoring: The overall score is a weighted sum of 10 categories. See our email archiving benchmark category scores for the weights and scoring rationale.
See our email archiving benchmark methodology for the weights and scoring rationale.
Ingestion and search tests
Ingestion coverage
We sent 10,000 synthetic emails through Microsoft 365 SMTP AUTH on a single day. Each vendor’s archive was checked for the full corpus at the end of the benchmark window.
Microsoft 365 transport silently dropped about 46% of the intra-tenant bulk submissions before recipient inbox delivery: 5,435 of the 10,000 mails reached the recipient inbox. NinjaOne and Barracuda capture mail at submit time (Sent Items mailbox-pull or journal-rule push), so the inbox-side drop does not affect their coverage. Piler reads only the recipient inbox; the 46% that Microsoft 365 quarantined never reaches Piler and never enters the archive. Piler’s inbox-forward architecture captured 100% of the 5,435 mails that did reach the inbox, but its coverage of the 10,000-mail corpus is 54%. We score Piler on the same 10,000-mail baseline as the commercial vendors; the gap is an architectural property of the inbox-forward path, not a Piler engine fault.
Acronis’s 80% coverage reflects what we observed when the benchmark window closed. The remaining 20% was still in progress at the end of our measurement.
Search latency and recall
We ran subject-keyword queries against each vendor’s search interface and recorded the result count and the response time observed in the UI.
Piler’s Manticore Search engine response time was measured directly against the engine; the commercial vendors’ response times include the admin-portal page render in addition to the engine. The recall column reports a qualitative match between the per-query hit count and the corpus ground truth count for that subject; we did not run a per-Message-ID cross-reference on Barracuda, Acronis, or Piler. On NinjaOne we did the per-Message-ID cross-reference through the EML export and confirmed full coverage.
Attachment recall and export tests
Attachment content recall
We embedded the marker phrase “Internal Document” inside every attachment across 8 file-type subtypes. For Piler, we ran each subtype in isolation (the test is straightforward when the engine output is directly queryable). For the three commercial products, we ran the aggregate “Internal Document” search with the With Attachment filter and recorded the hit count; per-subtype isolation was not run on the commercial UIs.
Aggregate attachment-marker hit counts on the three commercial products:
The corpus contains 1,700 attachment mails (600 PDF text + 100 PDF scanned + 300 DOCX + 300 XLSX + 100 PPTX + 100 PNG + 100 CSV + 100 ZIP). Send-side and recipient-side copies push the visible total above 1,700 in each vendor’s archive. The hit counts above are consistent with all 8 subtypes being indexed on each commercial product, but we did not run per-subtype isolation on the commercial UIs to confirm cell-by-cell coverage. Piler’s per-subtype test confirmed 4 of 8 coverage directly.
Export integrity
We ran the full 100-Message-ID integrity test on NinjaOne only. We downloaded the EML export bundle, parsed each EML, and verified body marker and subject preservation against the corpus. We did not run the same test on Barracuda or Piler within this benchmark; their bulk-export paths exist in the customer UI, but the per-Message-ID round-trip was not validated.
NinjaOne is also the only product that ships a vendor-side SHA256 checksum of the export bundle, shown next to the download link.
Acronis does not produce bulk EML or PST from the customer UI. Bulk actions on search results are limited to Send as email, Recover emails (re-injection into the original Microsoft 365 mailbox via Microsoft 365 transport), and Download attachments, with a cap of 10 emails per batch. For a single EML or PST handoff to outside counsel, the workflow requires recovering the mails to a temporary mailbox and then exporting that mailbox via Microsoft 365 Purview eDiscovery, two steps instead of one.
Compliance feature tests
Immutability
We attempted to delete an archived mail from each vendor’s customer UI using the standard tenant admin role.
The standard admin role cannot delete archived items from the customer UI on any of the four products. Deeper roles (partner-admin, super-admin) and storage-tier WORM tests were not run in this benchmark.
Legal hold
We exercised each vendor’s legal hold workflow by creating a hold on the user1 mailbox.
Three vendors offer working legal hold on the standard admin role. Barracuda requires the customer to flip the Litigation Holds toggle in Policy > Retention before the hold workflow is usable. Piler’s OSS 1.4.8 has no legal hold workflow in the customer UI.
Audit log
We exercised each vendor’s audit log by performing admin actions (create plan, create retention rule, create legal hold, run search) and checked log coverage and indexing latency.
Barracuda’s audit log was the most useful for compliance reviewers in this test. Every search query was logged with its full text and source IP. NinjaOne’s audit log was rich but some same-session events were not visible at +30 minutes in our window.
Encryption and BYOK
Acronis is the only product among the three commercial vendors that offers customer-set encryption in the customer UI. The password becomes the per-archive encryption key; recovery requires the same password. Piler’s encryption posture depends on where the customer hosts the Docker stack and how the storage is configured.
Retention granularity
NinjaOne’s item-level retention tag is the most granular pattern observed: every archived message carries an explicit retention badge visible in the search results. Acronis and Barracuda achieve per-user scope indirectly through saved-search filters (From:user@domain), which is flexible but requires customers to build and maintain the queries. Piler’s OSS 1.4.8 only supports a global retention setting.
Vendor access posture
We cataloged every Microsoft consent permission requested by each vendor at install time and every artifact created by the vendor inside the tenant.
NinjaOne’s 22 permissions span mail, SharePoint, OneDrive, Teams, calendar, contacts, OneNote, directory RBAC, and Azure AD application management. The same multi-tenant app is consented even when the customer activates only the Archiver tier. Acronis’s 10 permissions are mail-focused with three exceptions (Send mail as any user, Read and write all groups, Export user data). Barracuda’s 3 permissions cover all mailbox reading through one Exchange Web Services scope plus two minimal Graph permissions. Piler has no vendor consent surface because it is self-hosted; the customer-written ingest forwarder uses whatever Graph scope the customer grants.
Benchmarked email archiving products
The four product sections below document the full hands-on benchmark. All were installed on the same Microsoft 365 tenant and tested with the same 10,000-mail corpus, 1,700 attachments across 8 file-type subtypes, and the same 10 scoring dimensions.
NinjaOne SaaS Archiver
Onboarding
NinjaOne’s email archiving is bundled inside SaaS Backup. The customer navigates to Administration > Apps > Installed, enables the SaaS Backups card, then picks the SaaS Archiver plan tier (the other options are SaaS Backup, SaaS Archiver + Entra, and SaaS Backup + Entra). The plan tier cannot be downgraded once enabled.
After Enable, NinjaOne redirects to a Dropsuite EU partner portal where the actual archive lives. NinjaOne SaaS Archiver is a white-labeled Dropsuite tenant (Reseller ID 1999-18, EU Frankfurt). The archive index, search engine, retention policy UI, eDiscovery surface, and legal hold workflow are all Dropsuite property.
Dropsuite then runs a 4-step provisioning sequence that creates a dedicated Microsoft 365 Global Admin user named backupadmint<random>@<tenant>.onmicrosoft.com in the tenant. The credential is shown to the customer once and is used by Dropsuite for Exchange Online Remote PowerShell access alongside the Graph OAuth grant.
Dashboard & UI
The operational interface is the Dropsuite partner portal in a web browser. The dashboard shows mailbox count, backup status, storage used, and message statistics. The left sidebar covers Dashboard, Advanced Search, System Status (Backups, Downloads, Restores, Migrates), Insights, and Compliance (eDiscovery, Alert, Tags, Retention Policy, Legal Hold, Audit Log, Review Process).
Policy configuration
Retention is set per workload (Email, OneDrive, SharePoint) and per archive at plan creation. Period options are Years, Months, Weeks, Days, and Unlimited. Per-message retention tags are visible as inline badges in search results. Custom policies override the default per workload but cannot be scoped per user or per group directly.
Ingestion
Hybrid: Graph mailbox-pull plus a Global journal rule that Dropsuite creates in the tenant. Initial tenant backfill took approximately 3 hours through Graph plus Remote PowerShell. Coverage at the end of the window: 100% of the 10,000-mail corpus.
Search
Search runs through the Dropsuite Advanced Search interface in the partner portal. Subject, sender, recipient, body, attachment content, date range, and saved-query support are all available. UI latency was 1 to 2 seconds per query. Recall on subject-keyword queries was 100%.
Attachment recall
All 8 attachment subtypes were indexable. Scanned PDFs trigger OCR. XLSX, PNG, and CSV plaintext were all surfaced through the “Internal Document” marker search.
Export
Bulk export from search results in EML or PST format. A vendor-side SHA256 checksum of the export bundle is shown in the download UI, the only product in the comparison to ship one. Custody checklist 5 of 5.
Access control & AD
Standard admin role and Microsoft Entra ID integration through the multi-tenant app. Dropsuite-managed user accounts handle archive access; M365 group-based scope is achievable via saved-search filters rather than direct AD group binding.
Integration
Three SIEM export channels: Syslog, SNMP, and SMTP. Webhook subscriptions are not exposed in the customer UI. NinjaOne RMM integration is the natural fit for the customer’s wider ops stack.
Logging & alerting
Compliance > Audit Log surfaces three categories (Messages & File, User Activity, System). Date, user, and archive-ID filters plus export are available. We observed that some same-session events did not appear in the audit log at +30 minutes during our window.
Acronis Cyber Protect Cloud Email Archiving
Onboarding
Acronis’s Email Archiving module lives under Devices > Add > Applications in the Acronis Cyber Protect Cloud console, alongside the M365 Backup product. The two are distinct SKUs with separate Graph apps and separate consent flows.
After admin consent on the Acronis Archiving Service Graph app, the customer creates an archiving plan. Time to first archived mail was about 5 minutes after consent.
Dashboard & UI
The interface is a web console with sidebar navigation (System Status, Insights, Compliance) and a saved-search-query model for ad-hoc archive queries. Each saved query can be re-run, exported, or used to scope retention and hold rules.
Policy configuration
Schedule “Continuous archiving” is the default, and this is the only product in the comparison offering a continuous ingestion mode. The Encryption row in the plan create form carries a “Specify a password” link that activates BYOK; the password becomes the per-archive encryption key.
Retention is by saved search query (per-user, per-subject, per-date, per-attachment, per-size). Period options are indefinite or by age.
Ingestion
Graph webhook subscription for near-real-time capture plus Graph reads for initial mailbox enumeration. The benchmark window closed before the backfill completed; coverage at that point was 80% of the 10,000-mail corpus.
Search
Search runs through a saved-search-query form with sender, recipient, subject, body, attachment content, date range, deletion status, import status, attachment-only filter, and size range. UI latency was 1 to 2 seconds per query. Recall on subject-keyword queries was 100% on the backfilled subset.
Attachment recall
All 8 attachment subtypes were indexable on the backfilled portion of the corpus. Scanned PDFs trigger OCR.
Export
Acronis does not produce bulk EML or PST from the customer UI. Bulk actions on search results are limited to Send as email, Recover emails (re-injection into the original Microsoft 365 mailbox), or Download attachments (limited to 10 emails per batch). For compliance teams expecting a single EML or PST handoff to outside counsel, the workflow requires recovering the mails to a temporary mailbox and then exporting that mailbox via Microsoft 365 Purview eDiscovery, two steps instead of one.
Access control & AD
Microsoft Entra ID integration through the Acronis Archiving Service Graph app. Per-user scope on retention and hold rules is achievable through saved-search filters. The Acronis Cyber Protect Cloud platform supports custom roles and scope-based admin delegation.
Integration
Acronis Cyber Protect Cloud is a wider platform that includes backup, security, RMM, and email archiving as separate modules with shared identity. Webhook subscriptions are surfaced through the Microsoft Graph permission. SIEM integration is available through the platform’s reporting tier.
Logging & alerting
The Activities log surfaces admin actions in real-time, including plan apply, retention/hold rule execution, and admin sign-in. Mail detail action menu does not include a Delete option for the standard admin role.
Barracuda Cloud Archiving Service
Onboarding
Barracuda’s signup is an Email Protection bundle that includes Cloud Archiving Service plus four other email-security modules. Cloud Archiving Service does not show a Microsoft 365 Global Admin required badge on the bundle landing page; the path is journal-rule push rather than full OAuth admin consent.
The setup is in two parts. First, a customer-runnable PowerShell script creates a journal rule, an outbound connector, and a remote domain in the customer’s Microsoft 365 tenant, all pointing at Barracuda’s smarthost endpoint. Second, a separate OAuth flow under Mail Sources > Exchange Integration > Email Import enables EWS-based historical backfill.
Microsoft has announced Exchange Web Services retirement on Exchange Online in October 2026. Barracuda’s EWS Email Import path will need a replacement API before the deadline; the ongoing journal-rule push path is unaffected.
Dashboard & UI
The Cloud Archiving Service dashboard opens at zero. Account Status panel shows Directory Synchronization, Platform (Public Cloud), and Last Message Archived. The setup wizard at the top of the page guides the customer through Local Domains, Retention, and Apply Changes.
Policy configuration
Default retention is Keep emails indefinitely. Custom retention rules are scoped by saved search query, which can target per-user (From:user@domain), per-subject, per-date, per-size. Period options are indefinite or by age.
Ingestion
Journal-rule push for ongoing capture plus EWS Email Import for historical backfill. Historical backfill through EWS took roughly 30 minutes, the fastest in the benchmark. Coverage at the end of the window: 100% of the 10,000-mail corpus.
Search
Search runs through the Basic > Search interface and saved-search queries. Subject, body, attachment content, sender, recipient, date, and size fields are all available. UI latency was 1 to 2 seconds per query. Recall on subject-keyword queries was 100%.
Attachment recall
All 8 attachment subtypes were indexable. Scanned PDFs trigger OCR. The “Internal Document” marker search returned hits on every subtype.
Export
Bulk export from search results in EML or PST format. No vendor-shipped checksum. Custody checklist 5 of 5.
Access control & AD
Barracuda Cloud Control supports Microsoft 365 SSO for archive admins. Per-user scope on retention and hold rules is achievable through saved-search filters. Microsoft Entra ID group binding is not direct.
Integration
Syslog forwarding is available. Webhook subscriptions are not exposed. The cleanest non-app integration mechanism is the customer-managed journal rule itself, which can be audited and removed by the customer at any time.
Logging & alerting
Advanced > Audit Log surfaces four categories (Sign In, Search, Message View, Configuration Change). Each search query is logged with its full text and source IP. Date, user, and type filters plus export are available. We did not observe any indexing latency in our window.
MailPiler 1.4.8 (OSS)
Onboarding
MailPiler is open-source software packaged as a docker-compose stack (sutoj/piler:1.4.8 image with Manticore Search, MariaDB, and Memcached). We ran the stack on a Hetzner CPX22 VPS (2 vCPU, 4 GB RAM, Nuremberg). Time to first archived mail was 2 to 3 hours, covering Docker stack provisioning plus the customer-written Graph forwarder setup.
Dashboard & UI
Search runs through the Piler web UI on port 443 of the VPS. The interface is functional and OSS-typical: search box, result list, per-mail detail view. There is no commercial-grade dashboard for legal hold, audit log, or compliance reporting.
Policy configuration
Retention is set globally through a SQL retention setting in MariaDB. There is no per-workload, per-user, per-group, or per-message retention pattern in OSS 1.4.8. No template library.
Ingestion
Customer-written Python forwarder reads user1’s Inbox through the Microsoft Graph API using a customer-provisioned OAuth token, then relays each mail through an SSH tunnel to Piler’s SMTP listener on the VPS port 25. Every operational responsibility (token refresh, SSH tunnel health, certificate renewal, docker-compose upgrades) is customer-owned. Coverage of the 10,000-mail corpus: 54% (5,435 of 10,000). The shortfall is an architectural property of the inbox-forward path: Piler reads only the recipient inbox and cannot see the ~46% of mail that Microsoft 365 silently quarantined before delivery. Of the 5,435 mails that did reach the inbox, Piler captured 100%.
Search
The Manticore Search engine handles full-text and structured queries. The p95 query response time at the engine layer was under 90 milliseconds, the fastest in the benchmark. Recall on subject-keyword queries was 100%.
Attachment recall
Piler covered 4 of 8 attachment subtypes: PDF text-layer, DOCX, PPTX, and ZIP container unwrap. Missed: scanned PDF (no OCR pipeline), XLSX (Manticore does not parse Excel content), PNG (no image OCR), CSV (plaintext not surfaced by the body indexer).
Export
Per-mail EML export through a MariaDB retrieval script. No bulk export wizard, no PST output, no vendor-shipped checksum. The custody checklist is 3 of 5; operator ID and query text are not captured as a single audit record.
Access control & AD
No native Microsoft Entra ID integration in OSS 1.4.8. Access is managed at the operating system, web server, and database layers.
Integration
No SIEM forwarding, no webhook subscriptions, no built-in alert channels in the OSS stack.
Logging & alerting
Container-level logs from Manticore, MariaDB, and Piler-php. There is no customer-facing audit log surface, no compliance reporting, and no legal hold workflow in the OSS UI.
Email archiving benchmark methodology and environment
We provisioned a single Microsoft 365 E5 tenant and connected each vendor sequentially.
Sequential testing: each vendor was connected to the same tenant one after another. Vendors were not run concurrently because concurrent runs would have generated journal-report storms and conflated D1 measurement.
Test corpus
The synthetic corpus is 10,000 mails. 8,300 mails carry no attachment and 1,700 carry one attachment each across 8 file-type subtypes.
Send transport: SMTP AUTH at smtp.office365.com:587, user1 as the AUTH user, Microsoft 365 SendAs delegate enabled, rotated From header among the 25 shared senders. Send rate was limited by Microsoft 365 to roughly 30 mails per minute; the 10,000-mail send took about 5 hours of wall-clock time. After transport, Microsoft 365’s intra-tenant silent quarantine dropped about 46% of the corpus between Sent Items and Inbox: 5,435 of the 10,000 mails reached the recipient inbox.
Test scenario examples
Coverage test (ingestion). After the 10,000-mail send completes, log into each vendor’s archive UI and run a subject keyword search like “performance review” (corpus has 311 mails with this subject). Count the results and compare to ground truth.
Search recall test. Run 50 saved-search queries against each vendor’s UI. Each query targets a known subset of the corpus (subject keyword, sender filter, date range). Record the result count and compare to the corpus ground-truth count. A product with 100% recall returns every matching mail; misses indicate index gaps.
Attachment content test (subtype). Search for the marker phrase “Internal Document” in attachment body content with the With Attachment filter. Each attachment subtype is checked separately: if PDF text-layer files return hits, the product indexes PDF text. If scanned PDF files return hits, the product runs OCR. If XLSX files return hits, the product parses Excel content.
Export integrity test. Select 100 target Message-IDs from the corpus, export them through the vendor’s bulk export feature in EML or PST format. Verify body content, subject, sender, and recipient preservation. Verify the 5-item custody checklist: timestamp of export, operator ID, query text used, result count, and export format.
Immutability test. Log in as the standard admin role. Open an archived mail in the customer UI. Check whether the action menu offers a Delete option. If yes, attempt deletion and verify whether the storage tier respects or refuses the action.
Legal hold test. Create a hold on the user1 mailbox using the vendor’s hold workflow. Apply a retention policy that would normally purge mails after a short period. Verify that mails covered by the hold survive the retention purge.
Audit log test. Perform a series of admin actions (sign in, run search, create retention policy, create legal hold) and immediately check the audit log. Record which events appear, with what detail, and how soon. Re-check at +30 minutes for any indexing latency.
Email archiving benchmark category scores
Each category has a maximum score equal to its assigned weight. The weights sum to 100, so the raw total is the final score. Scores are integers derived from measured results or the documented feature matrix, not subjective ratings.
Export integrity contributes the largest single gap among the three commercial products (+5 NinjaOne over Acronis). NinjaOne is the only product with a vendor-shipped SHA256 checksum on the export bundle; Acronis is the only product without bulk EML or PST export. Ingestion coverage is the largest single gap between the commercial vendors and Piler (+9 NinjaOne over Piler), reflecting the 100% submit-time capture vs Piler’s 54% inbox-forward capture. Attachment content recall adds another +7 NinjaOne over Piler (8 of 8 subtypes vs 4 of 8). Vendor access posture is a +1 swing toward Barracuda over NinjaOne, reflecting the 3-permission consent surface vs the 22-permission consent surface.
Barracuda gives up 1 point to NinjaOne on D6 (Litigation Holds is off-by-default and the customer must enable it). On D9, NinjaOne’s item-level retention tag and Barracuda’s saved-search scope both land at the same 3 of 4 score. Acronis carries the only D8 = 3 in the comparison (BYOK customer-set key), contributing 2 points that the other three miss.
Scoring methodology (weights rationale)
10 categories scored on integer scales proportional to their weight. Each category’s maximum possible score equals its weight value. Weights sum to 100, so no scaling is needed. Weights were assigned based on which categories produce meaningful differentiation between products and which directly correlate with the compliance and operational concerns email archiving customers face.
Every score is derived from measured results (ingestion coverage, search recall, attachment hits) or from the documented feature matrix in the Feature comparison section, not from subjective assessment.
Feature comparison
Setup & deployment
Barracuda has the fastest backfill in the benchmark through EWS. Acronis’s backfill mechanism is continuous through Graph webhook subscription but did not complete in our window. NinjaOne and Acronis have the wider vendor footprint on the customer’s Microsoft 365 tenant.
Search & export
NinjaOne and Barracuda are the only products that produce bulk EML or PST output directly from search results. NinjaOne ships a vendor-side SHA256 of the export bundle, a chain-of-custody artifact none of the other three provide.
Compliance feature set
Acronis is the only product offering customer-set encryption keys in the customer UI. NinjaOne is the only product offering per-message retention tags visible at search time. Barracuda has the strongest audit log coverage and the only audit log we observed with consistent real-time indexing.
Limitations
Single Microsoft 365 tenant. This benchmark used a single Microsoft 365 tenant. Tenant age and policy configuration can affect transport behavior; new tenants have stricter Security Defaults than mature tenants. Results may shift on production paid tenants with different policy configurations.
English-language corpus. All synthetic mails and attachment markers are in English. Non-Latin script encoding (Arabic, Chinese, etc.) was not tested separately; some vendors document language-specific indexing differences that we did not exercise.
Cite this research
Pick the format that matches where you're publishing. Pasting the link version into your CMS preserves the backlink.
@misc{sar2026,
author = {Sarı, Ekrem},
title = {{Email Archiving Software Benchmark}},
year = {2026},
month = jun,
howpublished = {\url{https://aimultiple.com/email-archiving-software}},
note = {AIMultiple. Retrieved June 12, 2026}
}
Be the first to comment
Your email address will not be published. All fields are required. Comments are left in their original language.