Xamarin is a popular cross-platform mobile development environment, supported by Microsoft. It uses C# as a single language to create apps for Android, iOS, and Windows. The main reason to opt for cross-platform development – and Xamarin in particular – is sharing business logic and code-sharing across platforms including database access and network communication. This significantly reduces engineering effort that is required from the developers’ side to make UI code from scratch separately for each platform. Check our article on technical differences between cross-platform and native app development to get a fuller understanding.
But what about performance? It’s a well-known concern. Are cross-platform apps significantly slower than native ones? And if they are, would average users notice the difference? We’ve conducted a research to address these questions and understand what’s the real performance gap.
Another barrier to app performance testing is server-side operations. While you can have a perfectly-built application, most performance-related problems occur due to Internet connection and back-end architecture that streams data to apps.
To eliminate these inconsistencies, we’ve built multiple identical applications with basic features to compare them against each other on the same devices.
We also didn’t test on Windows phones, as they don’t have enough market share and seem less relevant to most app providers.
Xamarin.iOS & Xamarin.Android. The environment focuses on sharing business logic but configuring specific UI for different platforms. While this reduces the code-sharing capabilities for your developers, you can tailor UI to its platform to reach native app behavior.
Xamarin.Forms. This is a relatively new product that allows for sharing UI components to streamline development even more. The drawback to this approach is its less platform-specific behavior. Currently, Xamarin.Forms are actively applied for prototyping, making internal enterprise apps that don’t reach the mass market, or for simple, short-term apps. For example, you can make an app for a single event that won’t be updated and used further. Consider reading our Xamarin use cases story to know more.
We used both environments in our tests, except for photo loading and saving with Xamarin.Forms. All Xamarin tools were updated to the latest versions through Visual Studio version 15.3.
Native applications were developed with Java and Objective-C for Android and iOS respectively.
All performance measurements are made in milliseconds.
Package Sizes
While the speed difference between native, Xamarin Native, and Xamarin.Forms is certainly expected, the most drastic difference is noticeable across platforms. iOS performance generally exceeds Android performance for all tests. The boot speed is the quickest for Native.
The results are almost identical for all Android applications and don’t suggest that any of the engineering tools would be beneficial for apps that use multiple REST APIs. Surprisingly though, Xamarin.Forms Android app appeared 9 percent faster in these tests than even native Java.
iOS apps, on the other hand, show more predictable and expected results, as native appears to be the fastest one, Xamarin.iOS is 48 percent slower, and Xamarin.Forms is 64 percent slower than the native.
Deserialization is needed because the app may be written, for instance, in Java and the server-side architecture runs on C#: The object first needs to be serialized into bytes and further deserialized to let the C# architecture read it. The process of serialization/deserialization happens every time the app connects to a server through the API. As most modern mobile apps constantly create server connections, we considered serialization/deserialization to be a standard procedure that should be measured in terms of performance.
We used the Newtonsoft Json.net framework to serialize and deserialize JSON objects in all Xamarin apps. Also, we tested Native Android serialization and deserialization using the two most popular Java libraries: Jackson and GSON. They have their advantages and disadvantages in terms of performance, so we decided to include both in our tests. For the example of a JSON object, we used the one from OpenWeatherMap API, which is the London weather forecast.
The test is split into the first and the second run. During the first run the app performs serialization/deserialization, which clearly takes some time. During the second run, however, after the object has been cached, the process is performed significantly faster.
Both for serialization and deserialization of the first run, the performance pattern on all iOS applications is similar to the previous test. But this time, Native iOS is 247 percent faster than Xamarin.iOS in serialization and 1116 percent faster in deserialization! The difference between Xamarin.iOS and Xamarin.Forms isn’t that drastic, 42 and 105 percent for serialization and deserialization respectively.
Things aren’t that simple for Android. Xamarin.Forms and Native GSON-based serialization are almost equal in speed, while Native Jackson serialization is the slowest one: 231 percent slower than Native GSON and 54.6 percent slower than Xamarin.Android. We observed a different picture for deserialization. Native GSON remained the fastest one, but Xamarin.Android delivered the slowest this time, coming close to one full second to perform the operation.
The results of the second run are even more interesting for Android, as both Jackson and GSON native apps appeared to be much slower than their cross-platform analogs.
To sum up serialization/deserialization tests, all iOS apps keep the obvious pattern. Objective-C code is always faster than Xamarin.iOS and the latter is always faster than Xamarin.Forms. But if you look at actual numbers, you’ll see that even for such a large JSON object as we’ve chosen, iOS performs well on all platforms.
The Android results may seem unstable and too variable. However, the numbers are stable for given sets of observations. (As you remember, we provide the average of five observations per each feature.) For instance, the first run of deserialization on Xamarin.Android has a standard deviation of only 29.2 milliseconds. Native Jackson deserialization has a standard deviation of 47,8 seconds.
You may achieve some performance gains for iOS if you opt for native development. Quite surprisingly, with Android you will neither win nor lose in terms of serialization and deserialization. However, compared to native, Xamarin wins in the longer run in serialization/deserialization.
While we tested five images, we’ll provide the data on three as they are sufficient for accurate interpretation:
Unlike the previous set of Android tests, this one maintains a clear pattern. There’s no significant difference in loading images to RAM between Xamarin.Android and native performance. The native app is only 13-14 percent faster. The larger difference is visible in image-saving operations. Native performance is 42 - 46 percent higher. In real-life situations, apps use mostly smaller images and the 868Kb size is a more realistic scenario for estimating the time frame.
iOS tests have shown that both for image loading and saving operations you won’t see a substantial performance difference. Xamarin.iOS performs even slightly better for all image sizes.
We tested two operations:
BulkInsert: Loading rows of data into a database table.
GetAll: Retrieving all data from the database.
Each test operated with large databases having 10,000 records. All operations were processed internally on devices.
Again, the Android test demonstrated controversial results. BulkInsert was about 50 percent slower with native code, while Xamarin.Android and Xamarin.Forms behaved almost identically with this operation.
GetAll, on the other hand, revealed drastic differences both between Native and Xamarin.Android (521 percent) and between Xamarin.Android and Xamarin.Forms (305 percent). It took more than 5 seconds for Xamarin.Forms to retrieve the data.
As anticipated, iOS native code was the fastest one for both operations. Like Android, Xamarin.iOS and Xamarin.Forms don’t differ much with BulkInsert.
GetAll in Xamarin.Forms appeared to be 867 percent slower than Xamarin.iOS, which confirms the opinion that Xamarin.Forms is still early in its development to work on computationally heavy operations. This is true both for Android and iOS.
However, it provides enough information to draw several conclusions:
But what about performance? It’s a well-known concern. Are cross-platform apps significantly slower than native ones? And if they are, would average users notice the difference? We’ve conducted a research to address these questions and understand what’s the real performance gap.
Testing approach: devices, apps, Xamarin tools, and measurements
The main challenge to any performance testing is the difference between mobile applications in their architectures, business logic, operating systems, and ultimately required engineering expertise. That’s why users usually subjectively judge performance when looking at apps. Moreover, Android and iOS have different UI guidelines. These exist to let users recognize common features across apps and have a smooth onboarding experience. If you follow them – which is highly recommended – the same apps look slightly different and behave in a slightly different manner, which in turn may affect test results.Another barrier to app performance testing is server-side operations. While you can have a perfectly-built application, most performance-related problems occur due to Internet connection and back-end architecture that streams data to apps.
To eliminate these inconsistencies, we’ve built multiple identical applications with basic features to compare them against each other on the same devices.
Devices
We tested Android apps on Samsung Galaxy A7:- Android OS version: 6.0
- Central-processing unit: Octa-core 1.9 GHz Cortex-A53
- RAM: 3GB
- Display resolution: 1920x1080
- iOS version: 10.3.3
- Central-processing unit: Dual-core 1.84 GHz Twister
- RAM: 2 GB
- Display resolution: 1334x750
We also didn’t test on Windows phones, as they don’t have enough market share and seem less relevant to most app providers.
Applications
The main criteria for configuring the features for our applications were to choose the most common functions for all modern applications. To reduce the noise in measurements, we built separate apps for each of the following features, native and cross-platform:- Basic “Hello World”
- REST API
- JSON Serialization/Deserialization
- Photo Loading
- SQL Database Insert and Get All
Xamarin tools
We've broadly discussed specifics of the Xamarin environment. In a nutshell, you can work with Xamarin in two main ways: Use Xamarin.iOS and Xamarin.Android (so-called Xamarin Native) or opt for Xamarin.Forms. How do they differ?Xamarin.iOS & Xamarin.Android. The environment focuses on sharing business logic but configuring specific UI for different platforms. While this reduces the code-sharing capabilities for your developers, you can tailor UI to its platform to reach native app behavior.
Xamarin.Forms. This is a relatively new product that allows for sharing UI components to streamline development even more. The drawback to this approach is its less platform-specific behavior. Currently, Xamarin.Forms are actively applied for prototyping, making internal enterprise apps that don’t reach the mass market, or for simple, short-term apps. For example, you can make an app for a single event that won’t be updated and used further. Consider reading our Xamarin use cases story to know more.
We used both environments in our tests, except for photo loading and saving with Xamarin.Forms. All Xamarin tools were updated to the latest versions through Visual Studio version 15.3.
Native applications were developed with Java and Objective-C for Android and iOS respectively.
Measurements
Each test, except for photo loading, was repeated five times. As these observations have an insignificant standard deviation, you’ll see the average results on the graphs.All performance measurements are made in milliseconds.
Testing results
As mentioned above, we’ve built simple applications per each feature and per each platform. Let’s look at the key results that were generated.Hello World
Hello World applications were created to evaluate the sizes of apps and measure start time, the period that it takes to boot the applications.Package Sizes
Native apps are significantly smaller due to Xamarin libraries, Mono runtime, and Base Class Library assemblies used in Xamarin-based apps.
While the speed difference between native, Xamarin Native, and Xamarin.Forms is certainly expected, the most drastic difference is noticeable across platforms. iOS performance generally exceeds Android performance for all tests. The boot speed is the quickest for Native.
REST API
This set of tests is aimed at measuring the time it takes for the app to send a request through REST API and receive the response back without further data processing. We used the OpenWeatherMap API for these purposes (to successfully access the API you must sign up for the API key). Since API performance relies heavily on the Internet connection – the bandwidth and current load with other web-processes – we measured performance in identical web-load circumstances.The results are almost identical for all Android applications and don’t suggest that any of the engineering tools would be beneficial for apps that use multiple REST APIs. Surprisingly though, Xamarin.Forms Android app appeared 9 percent faster in these tests than even native Java.
iOS apps, on the other hand, show more predictable and expected results, as native appears to be the fastest one, Xamarin.iOS is 48 percent slower, and Xamarin.Forms is 64 percent slower than the native.
JSON Serialization/Deserialization
Serialization is the process of turning JSON objects into sequences of bytes to store and transport them during client-server communication. JSON is a widespread data structuring format used in most mobile applications.Deserialization is needed because the app may be written, for instance, in Java and the server-side architecture runs on C#: The object first needs to be serialized into bytes and further deserialized to let the C# architecture read it. The process of serialization/deserialization happens every time the app connects to a server through the API. As most modern mobile apps constantly create server connections, we considered serialization/deserialization to be a standard procedure that should be measured in terms of performance.
We used the Newtonsoft Json.net framework to serialize and deserialize JSON objects in all Xamarin apps. Also, we tested Native Android serialization and deserialization using the two most popular Java libraries: Jackson and GSON. They have their advantages and disadvantages in terms of performance, so we decided to include both in our tests. For the example of a JSON object, we used the one from OpenWeatherMap API, which is the London weather forecast.
The test is split into the first and the second run. During the first run the app performs serialization/deserialization, which clearly takes some time. During the second run, however, after the object has been cached, the process is performed significantly faster.
Both for serialization and deserialization of the first run, the performance pattern on all iOS applications is similar to the previous test. But this time, Native iOS is 247 percent faster than Xamarin.iOS in serialization and 1116 percent faster in deserialization! The difference between Xamarin.iOS and Xamarin.Forms isn’t that drastic, 42 and 105 percent for serialization and deserialization respectively.
Things aren’t that simple for Android. Xamarin.Forms and Native GSON-based serialization are almost equal in speed, while Native Jackson serialization is the slowest one: 231 percent slower than Native GSON and 54.6 percent slower than Xamarin.Android. We observed a different picture for deserialization. Native GSON remained the fastest one, but Xamarin.Android delivered the slowest this time, coming close to one full second to perform the operation.
The results of the second run are even more interesting for Android, as both Jackson and GSON native apps appeared to be much slower than their cross-platform analogs.
To sum up serialization/deserialization tests, all iOS apps keep the obvious pattern. Objective-C code is always faster than Xamarin.iOS and the latter is always faster than Xamarin.Forms. But if you look at actual numbers, you’ll see that even for such a large JSON object as we’ve chosen, iOS performs well on all platforms.
The Android results may seem unstable and too variable. However, the numbers are stable for given sets of observations. (As you remember, we provide the average of five observations per each feature.) For instance, the first run of deserialization on Xamarin.Android has a standard deviation of only 29.2 milliseconds. Native Jackson deserialization has a standard deviation of 47,8 seconds.
You may achieve some performance gains for iOS if you opt for native development. Quite surprisingly, with Android you will neither win nor lose in terms of serialization and deserialization. However, compared to native, Xamarin wins in the longer run in serialization/deserialization.
Photo Loading & Saving (First load)
Many modern apps use images. So, we decided to test two basic image operations: loading an image into RAM and saving the image to internal storage. The tests cover the first load only as further caching streamlines the process.While we tested five images, we’ll provide the data on three as they are sufficient for accurate interpretation:
- Resolution - 858x569, Size - 868Kb
- Resolution - 2575x1709, Size - 8Mb
- Resolution - 4291x2848, Size - 28.9Mb
Unlike the previous set of Android tests, this one maintains a clear pattern. There’s no significant difference in loading images to RAM between Xamarin.Android and native performance. The native app is only 13-14 percent faster. The larger difference is visible in image-saving operations. Native performance is 42 - 46 percent higher. In real-life situations, apps use mostly smaller images and the 868Kb size is a more realistic scenario for estimating the time frame.
iOS tests have shown that both for image loading and saving operations you won’t see a substantial performance difference. Xamarin.iOS performs even slightly better for all image sizes.
SQLite Database BulkInsert/GetAll
The final set of tests is aimed at estimating how Xamarin Native and Xamarin.Forms perform against native code in database operations. SQLite is a popular database engine frequently used for mobile development.We tested two operations:
BulkInsert: Loading rows of data into a database table.
GetAll: Retrieving all data from the database.
Each test operated with large databases having 10,000 records. All operations were processed internally on devices.
Again, the Android test demonstrated controversial results. BulkInsert was about 50 percent slower with native code, while Xamarin.Android and Xamarin.Forms behaved almost identically with this operation.
GetAll, on the other hand, revealed drastic differences both between Native and Xamarin.Android (521 percent) and between Xamarin.Android and Xamarin.Forms (305 percent). It took more than 5 seconds for Xamarin.Forms to retrieve the data.
As anticipated, iOS native code was the fastest one for both operations. Like Android, Xamarin.iOS and Xamarin.Forms don’t differ much with BulkInsert.
GetAll in Xamarin.Forms appeared to be 867 percent slower than Xamarin.iOS, which confirms the opinion that Xamarin.Forms is still early in its development to work on computationally heavy operations. This is true both for Android and iOS.
Conclusions
This research isn’t intended to bring closure to the ongoing discussion: What is faster, cross-platform or native?However, it provides enough information to draw several conclusions:
- iOS confirms expectations. The common opinion is that the more “native” the application is, the higher its performance. Xamarin.Forms which shares most of the code with its Android counterpart, tends to be the slowest one. We wouldn’t recommend using it for computationally heavy operations.
- You won’t gain much performance with native Android applications. The principle that works for iOS doesn’t apply to Android. We haven’t gathered enough evidence to state that native Android apps are faster than identical cross-platform applications. However, this difference might be more significant if you were to compare full-scale applications.
- Xamarin.iOS and Xamarin.Android remain the most balanced options. The ability to share all business logic across platforms and make platform-specific UI achieves the balance between budget and performance. Your single cross-platform team will be able to create most applications and maintain a decent performance level among other corss-platfrom mobile tools like React Native and Ionic.
- Most likely, users won’t be able to tell the difference between native code and Xamarin.iOS or Xamarin.Android. Too many factors impact user experience. Low connection speed combined with the poor backend is the main source of annoyance among app users. Only a small portion of UX problems are related to installed apps directly. It’s worth mentioning that a human can perceive and recognize performance delays above 500 ms. We tested both simple and complicated operations and most of them stay within or below this benchmark.