Assessing Customer Experiences with an Advanced Mobile Crowdsourcing Benchmarking Methodology

MedUX’s mobile QoE crowdsourcing methodology provides a customizable and reliable framework for assessing customer experience at scale. Building on our long-standing expertise in mobile benchmarking, we have adapted our unique scoring methodology to leverage massive crowdsourced datasets, combining technical and user-centric parameters and aligning with international standards. This empowers Mobile Network Operators (MNOs) and National Regulatory Agencies (NRAs) to make data-driven decisions while promoting fair competition.

Regulatory authorities and telecommunications operators face the ongoing challenge of ensuring superior Quality of Experience (QoE) while advancing investments in new technologies such as 5G and 5G SA. Crowdsourcing adds a new dimension: it enables continuous, real-world visibility of how networks perform where users actually are, complementary to where probes or drive tests can go.

Our crowdsourced mobile QoE benchmarking is highly relevant for multiple stakeholders, including Mobile and Fixed network operators, device manufacturers, equipment and service providers, content providers, regulators, and end-users. It allows them to evaluate service, device, and network performance, compare providers, and pinpoint areas for improvement. By offering objective and transparent insights based on real user devices, our benchmarking promotes fair competition and enhances the overall quality of mobile and fixed digital services.

Setting up the Crowdsourcing Benchmarking Framework

Customer experience has become a core performance metric for MNOs, embedded into operational reporting, NPS and churn models, and market positioning strategies. At the same time, NRAs are shifting from purely technical Quality of Service (QoS) towards QoE indicators that reflect how users actually perceive network and service performance.

With our refined methodology, MedUX remains at the forefront of evaluating mobile end-user QoE—also referred to as customer-perceived network quality or network experience. Our approach captures both technical QoS parameters (throughput, latency, jitter, DNS resolution time, packet loss, radio indicators) and user-centric QoE indicators related to web browsing, media streaming, gaming, social networking, and other everyday applications. All of this is synthesized in the MedUX QoE Scorecard.

Our crowdsourced QoE benchmarking framework is flexible and customizable, adapting to the specific needs of each client. To ensure reliable and repeatable results, we follow a standardized methodology encompassing all key aspects of mobile QoE.

The benchmarking process involves:

Selecting target use cases: defining objectives based on stakeholder needs—such as marketing and positioning, network intelligence, regulatory monitoring, technology/vendor assessment, or consumer advocacy.
Choosing appropriate data collection tools: Utilizing an optimal mix of proprietary software and hardware tools to simulate real user behavior and capture authentic QoE insights. Our multi-platform approach integrates crowdsourcing, SDKs, and Web Agents alongside complementary data sources, such as drive and walk tests via proprietary network testing robots.
Defining key performance indicators (KPIs): Establishing metrics and test schedules aligned with the QoE assessment goals, including QoS KPIs (speed, latency, jitter, DNS, packet loss) and service-level QoE KPIs (page load times, video startup delay and stalls, gaming latency, social media responsiveness, etc.).
Executing testing and analysis: Running automated tests on defined populations and territories and then applying our scoring methodology to deliver a fair, transparent, and independent evaluation reflected in the MedUX QoE Scorecard and MedOS (MedUX Opinion Scores).

MedUX QoE Crowdsourcing Methodology

MedUX delivers a comprehensive and objective view of mobile network experience through a global crowdsourcing methodology. By embedding a lightweight SDK into real mobile applications used by millions, MedUX captures user experience directly from end-user devices, generating reliable insights that support regulatory, operational, and commercial decisions across the telecom sector.

Below we summarize the four core pillars of this methodology.

Real-world data collection

Data is collected through a combination of passive and active measurements, providing a multi-dimensional view of network performance.

Active measurements replicate typical user activities—such as web browsing, file downloads, video streaming, or online gaming—and are executed under standardized conditions to ensure comparability across networks and geographies. Crucially, tests are launched only when the device is idle and connected, so they do not interfere with ongoing user activity or create unrealistic performance scenarios.
Passive measurements capture ongoing network conditions and device behavior without generating additional traffic. These include signal strength, type of network connection, technology in use, cell transitions, and handovers.

Data is gathered across a wide variety of real-world conditions: urban and rural areas, indoor and outdoor environments, and users who are either stationary or in motion. This broad coverage ensures both contextual relevance and robustness of the dataset.

Data validation and processing

To guarantee the integrity and reliability of the dataset, all collected data passes through a rigorous multi-stage validation and processing pipeline:

Automatic detection and exclusion of anomalous or corrupted samples
Removal of duplicates and samples that violate temporal or spatial uniqueness thresholds
Identification and filtering of extreme outliers in both radio metrics and active test results, which may arise from atypical environmental or device-specific conditions
Discarding results from devices running outdated operating system versions that could introduce inconsistencies in performance measurement

These validation steps are part of a standardized methodology aligned with international best practices and ITU-T recommendations, ensuring that MedUX performance analytics remain consistent, comparable, and technically sound across markets. The validated data is then processed in the MedUX Cloud, where KPIs are computed and transformed into QoE indicators and scores in near real time.

Representativeness and scale

MedUX’s global panel of tens of millions of devices enables a robust and statistically representative view of mobile network performance at national, regional, and operator levels.

For each country analyzed, MedUX ensures that the number of samples and unique users meets the statistical conditions required for robust reporting. As a general rule, key performance indicators are calculated at national level with a 95% confidence level and a maximum margin of error of ±5%, based on sample size and distribution.

While precision may naturally be lower for more granular splits (such as specific regions or technologies), the methodology guarantees that broader national comparisons remain statistically meaningful. Thanks to the continuous, high-volume nature of data acquisition, MedUX delivers near real-time visibility into connectivity trends, degradations, and improvements.

Privacy and data security

User privacy is a foundational principle in MedUX’s approach and underpins every aspect of data collection and processing.

No personally identifiable information (PII) is collected, stored, or analyzed at any stage.
IP addresses are systematically anonymized or aggregated, and all datasets are pseudonymized to prevent any possibility of re-identification, even in large-scale analysis.
The SDK runs seamlessly in the background with minimal impact on battery life, data usage, or device performance, and it requests only the permissions strictly necessary for measurement purposes, under a transparent and consent-based process.
MedUX fully complies with GDPR and all applicable local data protection regulations. Internal procedures are regularly audited, and data is stored and processed using secure infrastructure.

This combination of technical and organizational measures not only ensures compliance but also builds trust with users and partners by guaranteeing the ethical handling of network performance data.

Combining International Standards and Best Practices into a Unique Crowdsourcing-Based Scoring Methodology

The MedUX QoE Scorecard remains a trusted benchmarking tool for assessing network quality and performance. Our methodology continues to prioritize the end-user perspective by relying on real commercial devices in real environments, with real services and endpoints—now at crowdsourced scale.

The score integrates diverse KPIs across service categories, weighted according to international best practices and MedUX’s industry experience. Our evaluations encompass all elements affecting user experience, from environmental conditions and device capabilities to network infrastructure and service configurations.

Our methodology includes active speed tests that measure true end-to-end throughput via multiple Content Delivery Networks (CDNs). Operators may deploy dedicated measurement endpoints for specific projects, but not for independent benchmarking initiatives, preserving neutrality.

By adhering to international standards, including ETSI TR 103 559 and relevant ITU-T recommendations, we ensure a consistent, reliable, and comparable benchmarking process across operators and markets. At the same time, we review criteria and thresholds annually to reflect technological evolution, market dynamics, and evolving consumer behaviors.

Key principles of our scoring methodology include:

International standards and recommendations
Statistical robustness and comparability
End-user perspective
End-to-end performance
Real services and true QoE KPIs
Technology and market relevance

Leveraging True QoE KPIs from Crowdsourced Real-Field Data

Our crowdsourcing-based QoE benchmarking campaigns focus on the KPIs that best reflect customer experience with real services. By combining passive and active measurements on real user devices, we capture a multi-dimensional view of network performance across technologies (2G to 5G+), locations, and usage contexts.

Our evolving service experience categories, which can be reported separately for 5G-only or blended (3G/4G/5G) results, include:

Overall QoE Score: An aggregate of availability, accessibility, speed, streaming, and OTT experience results.
Reliability: Consistency of network performance across all categories, emphasizing stability over time and across locations.
Accessibility (responsiveness): Evaluates network access timespeed, latency, and responsiveness, emphasizing 5G network entry times.
Value for Speed (throughput): Assesses network capacity under stress, utilizing our patented Cloud Speed Test.
·Downlink Speed
·Uplink Speed
Video Streaming Experience: Considers video resolution, loading time, and stall rate.
Data & OTT Experience:
·Gaming performance (latency, packet loss, jitter) for popular online games
·Web browsing performance (loading times, DNS resolution, time to first-byte)
·Social media (considering session duration and time to content) for popular social media platforms

Each category is assessed through numerous KPIs, generating large datasets that are distilled into a synthetic performance indicator using our proprietary algorithm, grounded in international standards.

Exploiting our synthetic QoE indicator for actionable insights

The MedUX QoE Scorecard, and particularly the MedOS (MedUX Opinion Scores) for each service category, offers a simple way to translate dozens of technical KPIs into clear, easy-to-understand synthetic performance indicators. This allows mobile operators to understand the quality of experience their subscribers are having and to compare it against competitors.

The Scorecard is designed to be accessible across the organization, from executive management to engineering, optimization, and customer care teams. The service category scoring model and drill-down capabilities support granular analysis by operator, geography, technology, device type, or customer segment. Executives obtain a high-level view of performance and ranking, while technical teams can investigate specific issues at KPI level.

With this approach, providers and regulators can better understand customers’ QoE and take proactive measures to improve it. The MedUX Scorecard provides a reliable benchmark for mobile broadband performance, helps prioritize investments, and highlights the impact of new technologies—such as 5G on network performance and user experience. It also supports transparency and accountability in the telecommunications industry and helps ensure that consumers enjoy the best possible mobile experience.

Illustrating scoring principles across MedUX's Global Benchmarking Publication Series

MedUX performs a continuous series of QoE Crowdsourcing Benchmarks at the country level, based on the latest quarterly data, spanning key markets worldwide. These studies evaluate operator performance and overall network behavior, revealing how users perceive connectivity in their everyday conditions.

Based on our standardized methodology:

Select target use cases: Marketing and positioning, and network intelligence use cases.
Select appropriate collection methods MedUX Crowdsourcing solution, relying on our SDK embedded in real applications to collect passive and active measurements from millions of devices distributed globally.
Define key performance indicators (KPIs)
Data testing only (CDNs), web browsing (WhatsApp, Amazon, Google), social networks (Facebook, Instagram and TikTok), gaming (Epic games, LOL), streaming (Youtube), speed test, file download and upload, responsiveness and accessibility (ping), DNS resolution.
Conduct testing and analysis
Aggregate and validate millions of samples over each quarter and map all KPIs into the MedUX QoE Scorecard categories—Accessibility, Value for Speed, Streaming Experience, Data & OTT, plus overall QoE and Reliability—to produce operator rankings and awards such as Best QoE Experience and Best Streaming Experience for each country.

The above indicators are also calculated across an operator’s 3G, 4G and 5G networks with a 5G device and subscription to obtain a blended result across technologies for 5G enabled devices. In this case, it does factor the results in 3G, 4G and 5G networks, and also the availability of each network technology.

The overall score is calculated in a range of 0-5 points. Each service category is an aggregation of at least one MedUX Opinion Score. Each MOS is characterized by a Qualifier KPI and one or several Differentiator KPIs which turn technical measurements to a QoE metric of each Service.

In order to achieve an overall score, the individual test results have to be given a weight. This weight is the importance with which the result enters into the overall valuation of the service. The weighting of the results is done on each level of aggregation, from the service categories to the Qualifiers and Differentiators KPIs.

For more information don’t hesitate to contact us!

CONTACT US

Don’t forget to follow us on social media and subscribe to our newsletter to receive the latest updates and news.

About MedUX

MedUX is the leading Quality of Experience (QoE) company, providing comprehensive and innovative solutions for measuring the performance of fixed, mobile, and TV telecommunications networks for telecom operators, governments, and digital enterprises. With a focus on delivering grand scale, end-to-end network, and service visibility, MedUX leverages real-time customer perspective data and advanced analytics to ensure quality and regulatory compliance, while also offering valuable insights for optimizing networks and improving customer experiences. With a presence in over 25 countries across Europe, America, Africa, and the Middle East, and monitoring over 60 operators worldwide, MedUX’s patented technology and expertise make it a trusted partner for improving the digital experiences of customers everywhere.