Software Evaluation 5 Steps to Pick a Winner

This article is inspired by a task that fell to me recently: To recommend an adequate-to-perfect project management organization (PMO) solution for our company. I will deliver my recommendation soon, but even more interesting to me is analyzing the evaluation process itself and establishing a model for future exercises of the same nature.

Step 1. Make sure you see a vision behind the criteria.

Everyone will tell you that working out a list of mandatory and nice-to-have criteria is where you start. Many articles, books, and white papers are written on this subject. Here is what I can add to the discussion, based on my experience:

  • In an ideal situation, you need to care about the solution you are seeking because you will be dependent on the result of the search. At the very least, the subject of the research should be of keen interest to you or a true subject of your expertise. If none of the above is true, you should recognize that you are not the best person to do the research. While you might lend a hand in managing the process, you should engage another team member to perform the core evaluations.
  • Develop a vision! You need to “see” how things are going to be done to envision the right solution for you team or your client. The vision naturally leads you to the prioritized list of criteria. In other words, if you have a vision of what you really want to accomplish, the components of this vision automatically move to the top of the list of priorities.
  • Involve as few decision-makers as possible. The more folks you ask to be involved, the more conflicting opinions you will have, and eventually you will be convinced that the solution you are looking for simply does not exist. Do not let this happen! Be a visionary and an expert, or rely on the expertise of a very narrow circle of advisors.

Step 2. Cast a wide net.

This is the fun part. You start compiling a long list of potential solutions, most of which you have never heard of. And just when you think you’ve discovered them all, new candidates keep popping up.

This is probably the easiest part of this project, because you are not yet at the point when you need to make a conclusive decision. At this stage, it is all about keeping an open mind and rejecting any potential solution that does not align with your vision out of the gate.

Of course, if you are lucky, you will soon come across something that you fall in love with, or something that was very strongly recommended by someone whose expertise you trust. This did not happen to me, but may happen for you; if it does, my advice is to concentrate on that solution. Do not neglect the rest of the process, but do not be afraid to play favorites either. Chances are that your first instinct is the right one, and your job is easier than you thought it was.

But if the miracle does not happen early, you are back to the process of review and rejection. Look at websites and the list of features, and, even more importantly, look closely at screenshots and videos — those will speak volumes. Oh, and don’t forget to check YouTube: for each video tutorial developed by a vendor, you will probably find ten developed by end-users.

Here is where your vision should really work to your advantage. Not through the list of features alone (these will never tell you the full story), but through a combination of descriptions and visuals, will you be able to gauge the degree of match or mismatch between your vision and what the evaluated solution has to offer.

Is this superficial? You bet it is. But we are talking about a complex task and the reality of your busy workday, and you likely have very little time to work on this — both in duration and in effort. That means that you are not in a position to take a deep dive into every single product that comes your way. Besides, once you are armed with your vision, your instincts, and your expertise, holding your sword/budget in one hand and your shield/list of criteria in another, the rejections will come easy.

Step 3. Select a manageable number of potential candidates.

Ideally, you are now operating with a single-digit number of candidates. At this point, it is wise to start scheduling in-person presentations, or attend webinars if they are being offered. Once again, you will probably quickly reject some of the candidates. Often even a very brief, slightly deeper look leads to the realization that the solution is not the right one for you, even if it looked good on paper.

This is also the right time to start getting into the free trials and demos, and to initiate your blow-by-blow comparison list. This one will come in handy later.

A note on free trials: Some vendors will allow you to test their software and some will not. My recommendation is to never trust software you cannot try for yourself, unless we are talking about complex enterprise solutions that simply cannot be presented for hands-on evaluations. In my experience, some of the high-end vendors will prefer not to allow free trials, but may agree on granting access if you stay firm.

Step 4. Face-off.

Here it is. You are left with a few top candidates, and the most-difficult decision is still ahead of you, unless you managed to fall in love with one of the bachelorettes (oops… now you know what I’ve been watching on TV in my spare time). Now is the time for a true deep dive. Log in to or install your free trials, experiment as much as you can, and fill out your comparison grid.

In a few days, you will feel that you are becoming an expert with some systems, and are not getting anywhere with others. If this is the case, take it as a very important clue. If you see that expertise eludes you, it will be even worse for your users. And if you feel comfortable, it will be that much easier to help others down the road.

Most importantly, find a way to document your findings. If you don’t, it will all seem like a blur, where memories of one package will soon be replaced by another, depending on what you concentrate on at the moment. In my experience, the quickest way to take notes is to take lots of screenshots; those will quickly trigger your memory of what worked and what did not and why.

Step 5. Make a decision.

Finally, you will need to present your findings and justify your decisions to your clients or your stakeholders, and you will have to stand firm in your expertise, even if your memory is getting fuzzy. The final decision may not be up to you to make, but you should always have a strong opinion, and be able to defend it. After all, if you do not know what to choose after all this research, who else will?

Just saying…