A repository has thousands of stars. The demo looks excellent. Your team cannot restore its database.
That last fact deserves rather more attention than the first two.
The short answer: evaluate open-source software against your workflow, rights, security, operating capability and whole-life cost. Use scores to compare evidence, with hard gates for unacceptable risks. Popularity is a discovery signal, not an acceptance test.
Define the job before comparing products
Write down the users, essential tasks, data, integrations and service expectations. Separate requirements from preferences. A team that calls every preference essential has not yet prioritised.
Choose representative scenarios to demonstrate. For a case-management tool, that might include creating a case, restricting access, correcting a record, exporting it and restoring it after failure. Test the awkward cases as well as the demonstration account.
The software may be excellent and still be wrong for your organisation. A system designed for a large engineering team can become a burden for an institution with one administrator and no reliable support arrangement.
Score evidence, not confidence
Use 0 for no evidence, 1 for a credible claim, 2 for a demonstrated capability and 3 for a capability tested in your environment. These are suggested scoring rules, not an industry certification. Record the reason beside every score.
| Dimension | Suggested weight | Evidence worth seeing |
|---|---|---|
| Workflow fit | 3 | Your representative scenarios completed |
| Security and maintenance | 3 | Supported release, vulnerability process and patch rehearsal |
| Licence and rights | 3 | Exact release licence and dependency review |
| Operations | 3 | Installation, monitoring and successful restore |
| Privacy and data handling | 2 | Data-flow, telemetry, retention and deletion checks |
| Integration and export | 2 | Documented interfaces and usable export |
| Community and support | 2 | Maintainer activity and credible support options |
| Whole-life cost | 2 | Hosting, people, upgrades and exit assumptions |
With these weights, the maximum is 60 points. Divide the weighted result by 60 if you want a percentage, but keep the underlying evidence visible. A percentage is not a prediction of success.
Set hard gates before adding the scores
A candidate that lacks a required legal right or cannot protect restricted records should not win because its interface is attractive. Identify blockers before reviewing products, including requirements imposed by your organisation or sector.
Inspect privacy separately from security. Record which information leaves the deployment, including telemetry, logs and support uploads; who receives it; and how retention and deletion work. Test the relevant settings with representative non-sensitive records. A self-hosted product can still make outbound requests. Ask the data owner to resolve any applicable legal or contractual constraints before using real records.
For example, require proof of a usable data export and a supported security update route. A failed gate means reject, remediate or narrow the intended use. It does not mean quietly reducing the weight until the preferred product wins.
Use the security checklist for deeper inspection. The OpenSSF Scorecard provides automated signals about repository practices; it does not replace an assessment of your deployment.
Run a short, realistic evaluation
Install a pinned release in a disposable environment. Record what a competent administrator needed to do, which instructions were missing and whether external services were required. Import a sanitised sample, run the scenarios and attempt an export.
Then rehearse an upgrade and restore. Measure elapsed time and human effort. Invite the people who will use and support the system to identify workarounds. A technical evaluator can miss the daily burden of five extra clicks across hundreds of records.
Keep the environment separate from production and avoid uploading sensitive data merely to test a public demo. The evaluation should reduce risk, not introduce a new one.
Count the costs that remain after the licence fee
Include hosting, security work, implementation, migration, training, support, upgrades and the people who own the service. Ask what happens if a maintainer leaves or a support company changes its offer.
Compare at least a base case and a difficult case. The difficult case might involve a major upgrade, a custom integration rewrite or a supplier transition. Price uncertainty explicitly rather than hiding it inside “free”.
Make a decision someone else can inspect
Retain the version tested, scorecard, failed gates, evidence links and decision owner. Record what must be completed before deployment and what will trigger a reassessment.
The winning product is the one you can use and sustain for the job at hand. Sometimes that is open source. Sometimes the honest evaluation tells you to choose something else.
Sources and scope
Reviewed 1 November 2026. The examples and worksheets are my suggested working methods, not measured client results or official standards.
