Measuring Performance is Tricky
Measuring performance is simple to initiate – just run a speed test.
Nevertheless, there is a modest number of key factors that can impact speed test results that are often missed, not understood, or simply not known at the time of the test.
At the time of the test, a network test on a mobile device is dependent on a combination of factors including and not limited to
the behavior of the network,
any other activity on the device,
data plan details (if it is in use),
any quality of service settings between the device and the speed-test server,
raw transmission bandwidth,
power remaining in the device,
wireless bandwidth allocation strategies,
load on the speed test server(s),
device driver implementation details, and
how the test operates.
The preceding paragraph shows that no one, not even the best technologists, can fully explain a particular speed test result.
So where does that leave performance measurements, the purpose of which, is to confirm for the end-user that they are using the best tools?
To that end, the FeatherVPN team has developed the following test methodology for comparing FeatherVPN against other VPN’s
and against the native network.
- For each speed test service used to measure performance, complete items 2 to 8.
- Define a trial as an equal number of runs of the speed test.
- Use the same device for all tests and ensure that the operating system is not using the network prior to the trial run.
- Make sure that the test device is fully charged.
- Run a NO-VPN baseline trial using mobile, wifi, and host network.
- Run a VPN trial using mobile, wifi, and host network.
- Run all trials on same mobile network provider.
- Run all trials in the same location and if possible against the same test server.
- Run all trials sequentially back-to-back.
The idea is to make sure that as many as possible of the factors listed earlier that can impact a speed test result are the same for all trials,
thereby, the ensuing test results can be relied upon.
It is possible that tests conducted in a different location, or using a different devices,
or different host network providers, may differ.
When such an anomaly from predicted results is detected, a sound methodology makes it easier to investigate its cause.