As soon as the 2024 MITRE ATT&CK Evaluations results were released, some vendors started making questionable claims about achieving “100% scores,” relying on half-truths and, in some cases, outright misrepresentation.

Let’s break down the misdirection.

Common Misrepresentations

  1. Cherry-Picking Comparisons
    Instead of comparing themselves to all participants, some vendors carefully select competitors to highlight their superiority. In some cases, they even exclude smaller vendors who outperformed them or compare themselves to companies that did not participate.
  2. Selective Analysis and Misrepresentation of Results
    Some vendors showcase “100% scores” without proper context or disclaimers. Here are a few tactics they use:
    • Configuration Changes: Vendors request multiple retries (second, third, or even fourth chances) to fine-tune their solutions after initial misdetections, artificially improving their performance during re-executions.
    • Ignoring N/A Sub-Steps: N/A (Not Applicable) is assigned by MITRE for sub-steps that aren’t relevant—like those involving unsupported operating systems (e.g., Linux or macOS) or steps affected by technical issues during execution. However, some vendors misinterpret N/A as a “miss” in their analysis. Remember: N/A impacts both the numerator and denominator.
  3. Inventing Misleading Metrics
    New, unofficial metrics like “Best Signal-to-Noise Ratio!” or similar terms are being created to spin results in their favor. These metrics are not endorsed by MITRE and often have no basis in the evaluation’s methodology.

Below is a table displaying the raw results of the 2024 MITRE ATT&CK Evaluations before vendors applied Configuration Changes. This analysis provides a transparent view of the initial performance of various solutions.

Go beyond unsubstantiated vendor marketing claims for the facts. Leaders are strongly encourage to verify the results directly on MITRE’s website. It’s user-friendly, and you can compare participants based on your own criteria.

View results on MITRE’s ATT&CK Evaluation site

When visiting MITRE’s website, keep in mind that by default, results are displayed after vendors applied Configuration Changes. To view the raw results before Configuration Changes were applied, uncheck the “Configuration Changes” checkbox in the filters. This ensures you’re seeing the vendors’ initial performance.

What You Can Do

The real data speaks for itself. Don’t rely on flashy marketing claims or let misleading claims cloud your judgement. Look at the raw results directly. Transparency and accuracy matter when evaluating solutions for your organization.