Measuring Performance is Tricky
Measuring performance is simple to initiate – just run a speed test.
Nevertheless, there is a number of often missed key factors that can impact speed test results. These are commonly misunderstood or simply not known.
At the time of the test, a network test on a mobile device is dependent on a combination of factors including and not limited to
the behavior of the network,
any other activity on the device,
data plan details (if it is in use),
any quality of service settings provisioned between the device and the speed-test server,
overall network-latency,
network jitter,
raw transmission bandwidth,
power remaining in the device,
wireless bandwidth allocation strategies,
load on the speed test server(s),
operating system version,
device driver implementation details, and
how the test itself operates.
Methodology
The preceding paragraph shows that no one, not even the best technologists, can fully explain a particular speed test result.
So where does that leave performance measurements, the purpose of which, is to confirm for the end-user that they are using the best tools?
To that end, the FeatherVPN team has developed the following test methodology for comparing FeatherVPN against other VPN’s
and against the native network.
- For each speed test service used to measure performance, complete items 2 to 8.
- Define a trial as an equal number of runs of the speed test.
- Use the same device for all tests and ensure that the operating system is not using the network prior to each trial run.
- Make sure that the test device is fully charged.
- Run a NO-VPN baseline trial using mobile, wifi, and host network.
- Run a VPN trial using mobile, wifi, and host network.
- Run all trials on same mobile network provider.
- Run all trials in the same location and if possible against the same test server.
- Run all trials sequentially back-to-back.
The goal is to ensure that of the many previously listed factors impacting network speeds remain consistent over all trials.
Thereby ensuring that the obtained test results can be considered reliable.
It is higly likely that tests conducted at different locations, or using different devices,
or different host network providers, will differ. That is normal. The FeatherVPN team works extrodinarily hard to enable you to
most of the time experience performance results that few if any other VPN services can deliver.