An Evergreen platform can help you avoid running into compatibility issues that stop your Windows rollout in its tracks by automating testing. In this review I look at three platforms from AppAvail, Application Readiness, and Apptimized.
Application compatibility.
For anyone who’s worked in an End User Computing role in at least the last 10 years, application compatibility has been a challenge. The migration from Windows XP to Windows Vista or Windows 7, in particular, was very painful for many. The good news with initial releases of Windows 8/8.1 and Windows 10 is that application compatibility was much less of an issue. Microsoft took more care to ensure that if an app worked on Windows 7, it would also work on the newer desktop operating systems.
Fast forward to the present day. Microsoft has drastically changed the release cadence for their Windows desktop operating systems with Windows 10. While application compatibility still isn’t as big of an issue as it once was, it’s a challenge that has been returning. Security hardening out of necessity has lead third-party applications breaking and, in some cases, even some of Microsoft’s own products breaking too!
We have to accept that some upgrades are going to result in broken applications, for most organizations, that means pushing Windows 10 upgrades and just hoping for the best. Windows migrations of the past would take 18+ months on average to complete, and that was usually with a large project team assisting. In large part, the time and effort were spent on application testing, but it’s not feasible for day to day IT teams to use the same methodology for the new, more frequent OS upgrades. There just isn’t enough time and resources.
Evergreen Platform Review.
In looking at Evergreen platforms, I’m going to mainly focus on Apptimized, AppAvail, and Application Readiness. I’ll concentrate on the ability to use these platforms to automate your application testing to streamline your Windows 10 upgrade process, and what I feel are the strengths of each product.
AppAvail.
I must say that out of the three products tested in this article, AppAvail was by far the easiest to use. The workflow and UI were very intuitive.
Before uploading my existing packages from my organization to AppAvail, I needed to ensure each package was in a zip file first. This is something that was unique to AppAvail. Once I had my packages zipped up, it was a simple matter to upload them and fill in any metadata that may have been missing like the version, for example.
Each package will have its own install command. Assuming your package, like most enterprise deployments, can be installed via command line, the Evergreen test is simple. The product will spin up a VM, do the install, and launch each shortcut. It will also do some basic logging and even parsing to flag anything that might be an issue and let you know in the results. That way, you don’t have to babysit the process and can review only those that have warnings or errors when it’s complete.
As you can see in the above screenshot, the product also generates step by step screenshots of the install and test, as well as a handy compressed video. If there was an issue detected, you could simply look at the screenshots or videos to learn more about it.
A couple of the applications I tested were just a set of files without any package, some of the apps had scripted installs, many were MSIs. For those that I didn’t have a package for and were just a set of files, I simply selected to manually test on virtual (an option you can see in the above screenshot). I then determined if it worked or not and chose to override the results and set my own result for that app.
The end result was a list of my applications showing which ones tested successfully, which ones failed and which ones completed with warnings. I could then review if there were warnings or errors. If the video showed the install failed due to a missing pre-requisite or something of that nature, I could test manually and include the pre-requisites and then override the results to successful if it then worked.
If they worked manually, but I discovered the package needed to be tweaked, I could repackage it and test again, or if it failed completely, I could leave it as a failure and move on. The above screenshot is an overview of the results for Windows 10 1903.
But as you can see above, I also have results against 1809. You can have VMs for any and all versions and simply run the tests as you wish. In my testing, I was running tests on 1809 and 1903 for the same application at the same time. It was all very quick and very easy, and the end product list of my apps and their automated testing results were exactly what I was hoping for from the product.
Application Readiness.
Application Readiness had the most comprehensive application compatibility reporting of the three products. It was pretty different from AppAvail in how it performs the testing and in its level of reporting.
For comparison’s sake, following the same flow as with AppAvail, I started by uploading my packages. I uploaded all of my packages to Azure storage through Microsoft Azure Storage Explorer. There was no need to zip the packages in this case.
I then linked my Azure Storage to the service using the SAS Key and scanned the drive from the Application Readiness portal. There was also an option to use OneDrive shared folders for the upload.
As stated previously, my set of applications contained a mix of executables, scripted installs, MSIs, and even a set of files for manual installs. The scan auto-detected the installers and even associated any MSTs it found with the parent MSI.
Once I had the applications loaded, I queued the import of the apps.
Pretty quickly, the applications were loaded and ready to test.
This is where Application Readiness takes a big divergence from the other two products. Some of the underlying technology in Application Readiness was taken from ChangeBase. Part of which, can dissect an MSI package, look at its contents, and report on compatibility based on what’s found in the package. The apps I imported that were straight-up MSI could generate a RAG (Red Amber Green) status right away, showing if the app is going to have issues or not.
Now, above you see red and some .exe’s that don’t have a status. Don’t worry about the reds, the reports I ran were not just against Windows 10 versions, it was also against things like App-V 5.1, VMware WorkspaceOne, VMware ThinApp, AppX and even MSIX.
That’s right. Application Readiness can tell you if your organization’s existing packages are likely to work in many different package formats. Above, you can see an example report generated from me drilling into the RAG status of an app I imported. Looking at the red status for MSIX and what the reason provided is, we see that, in this case, it was a print driver that is currently not supported with MSIX.
For those apps that don’t return a RAG status right away, if there are embedded MSIs, the product will tell you, and you could then use those to report against. Otherwise, you can provide the install parameters for your .exe or whatever type of package it is to allow Application Readiness to automate the testing on a VM, much like with AppAvail. Like App Avail and indeed Apptimized, if needs be, you can manually install and test the app on the VM in the portal yourself.
For an example of some of my Windows 10 testing with Application Readiness, I show above some of my results for PhotoShop CS2, which is quite old, and as you can see, it found a 16-bit file which requires further testing from me.
Something to note that is really cool is how insanely quick it is to show you the RAG status for the environment and package type you care about specifically. So as you can see above, I select my Platform Desktop as going from Windows 10 1803 32-bit to Windows 10 1909 64-bit. I didn’t bother reporting on a specific version of .NET Framework, but I could have. I was interested in seeing how many of my apps would work as MSIX (not many as the reports would show 🙂 ), the Edge browser, and Office 2016, AND it’s so granular that I selected Feb 2020 to include reporting with the latest Windows Updates.
The reporting down to the security patch level is unique to Application Readiness and makes it the most comprehensive in this review.
Personally, I feel the current workflow in the product needs some work, but it was also improving in real-time over the few weeks I spent testing it. It’s still a pretty new product to the market, and based on my experience with ChangeBase in the past, I’ve no doubt the workflow will be improved upon.
Apptimized.
Apptimized had the most innovative method for determining testing success and failure, but before I get into that, one difference right off the bat compared to the other two products is that Apptimized has integrated logins for Microsoft accounts or Linkedin accounts, which was pretty interesting. Uploading my packages was also very simple. For uploading, it was the most straight forward of the three products.
Upload, browse to the packages, and it just performs an upload. Your packages are stored on Apptimized’s storage and ready for use.
You could also do this via the Add application button or the upload link. I must say the workflow in Apptimized is very good. You can see it for yourself in the above screenshot. The steps are plotted out for you, and best of all, you can skip directly to a step or in the event of having already completed a step, but wanting to re-do it, you can just go back to that step and start it again. It’s very simple.
Let’s get into how Apptimized is so innovative. The product has a feature called Echo. Above is a sample of some of the apps I ran through with Echo on against Windows 7 64-bit and Windows 10 1809 as an example, but you could test on different versions of Windows 10 too.
Here’s where this starts to blow my mind. The product does an automated application discovery, generating a discovery document that includes the install steps and launch behavior of each app. It then lines up the steps side-by-side, matching from one OS to the other to show any discrepancies. It also does Windows logging, and it’s own form of logging to capture any potential issues and show those in the overall success or failure of the testing.
If there is an issue or discrepancy detected, you have the option to re-run the automated test, or you could retest manually by completing the install and launch yourself in the VM provided in the portal. As packaging is possible, you could also try repackaging the application and instead test with the package OR something that’s rather nifty – you could submit to Microsoft as part of their Desktop App Assure offer to help investigate and remediate app compatibility issues on Windows 10.
Conclusion.
It’s hard to separate the products because they each have unique features and strengths. AppAvail is the easiest to work with, and it has a very flat intuitive UI and workflow. Application Readiness has more comprehensive reporting and has some great security benefits with its patch reporting. Apptimized is really innovative and has a nice UI and workflow.
My original idea for this post was to also compare based on the results of the reporting, but during my testing and research, I decided that wasn’t a very good way of comparing the products.
The reporting in Application Readiness is very different from the other products, and for a simple go or no go, I feel like the results were pretty close across the three products. When one showed failure, where the other showed success, I found it was for a reason that wasn’t a big deal, For example, one of the apps I picked was Camtasia and in one of the products the testing VM was the Windows N edition and Camtasia requires Windows Media Player to install. Technically it didn’t work, but when I tested on a full Windows 10 version, it did.
Also, I feel basing a decision on which product is supreme on its success vs. failure reporting for ongoing Windows 10 testing would be a little foolish as the majority of apps will be a success. The 20+ year old apps I picked to trigger failures are not reflective of most organizations Windows 10 1809 to Windows 10 1903 appset :-). I’m pretty confident each of these products will give you a good level of insight into the success rate of app testing when preparing for your next Windows 10 upgrade. Which product is best for you should be based on which features and reports are best for your needs.
Understanding the bigger picture.
Finally, it’s worth noting that while all the products reviewed can also offer assistance with some automated repackaging capabilities, as has been the case for as long as I have worked in packaging, mileage may vary, and for the greatest success rate, you’ll still want to have skilled packagers like Algiz Technology work with the tools and provide their expertise in helping you see the big picture of what’s happening with your application estate.
Oh, and be sure to sign up for alerts below for new articles like this one.
