---
title: Eleven questions to ask before you let software rank candidates
description: Every vendor says their screening is fair and explainable. These are the eleven questions whose answers you can verify — and the one that says walk away.
---

[Skip to content](https://www.echorecruit.com/blog/eleven-questions-to-ask-before-you-let-software-rank-candidates#main-content)

[![echo-recruit-logo-760x180](https://www.echorecruit.com/hs-fs/hubfs/echo-recruit-logo-760x180.png?width=300&height=71&name=echo-recruit-logo-760x180.png)](https://www.echorecruit.com)

Open main navigation

Close main navigation

- [Contact us](https://www.echorecruit.com/contact-us)

[Contact us](https://www.echorecruit.com/contact-us)

[All posts](https://www.echorecruit.com/blog/all)

 September 29, 2026

# Eleven questions to ask before you let software rank candidates

 By   Echo Recruit  ·   5 minute read

*You are not evaluating whether the model is good. You are evaluating whether, eighteen months from now, you can reconstruct why a particular person did not get a call.*

In short

- Ask what is deterministic and what is a model. If everything is a model, nothing is explainable.
- The single most revealing question is what happens to a requirement the CV cannot prove. A silent zero is a broken process.
- An evaluation should be a new row, not an edit. Overwriting means last week’s decision no longer exists.
- Automated employment decision tools are regulated in a growing number of places, and the obligations mostly fall on you, not the vendor.

Screening software has got good enough to be worth buying and confident enough to be worth checking. The demo will be impressive, because ranking a pile of CVs is genuinely something this technology does well. The demo is not the thing to evaluate.

The thing to evaluate is what happens afterwards, when a rejected candidate’s solicitor, a client’s procurement team, or a regulator asks a question about a specific decision made on a specific day. These eleven questions are the ones whose answers you can verify rather than take on trust, roughly in the order they become expensive.

## On how it works

### 1. Which part of this is arithmetic and which part is a model?

There should be a clear line. Arithmetic — does the candidate have these skills, this many years, a title that matches — is deterministic, repeatable and explainable to a non-technical person. A language model reading a CV and forming a view is none of those things, and it is also better at the parts arithmetic cannot do.

A good architecture runs the deterministic pass first and uses the model for judgement on top, with both visible. A vendor who cannot draw that line for you either does not have one or does not want you looking at it.

### 2. Can you show me the weights?

Not the model weights — the rubric weights. Skills 50%, experience 25%, role match 25%, or whatever they are. You should be able to read them, change them, and see the score move.

“Our matching algorithm is proprietary” is a complete answer to this question, just not the one being asked for. If you cannot see the weights, you cannot tell a hiring manager why one candidate ranked above another, which means you have outsourced the part of your service you are actually paid for.

### 3. What happens to a requirement the CV cannot prove?

The most revealing question on the list, and the one most likely to get a vague answer.

Salary expectation, notice period, visa status, willingness to be on site: none of these are on a typical CV. If the system scores them as zero when absent, it is systematically marking down everyone who writes a normal CV, and the people who float to the top are the ones whose documents happen to mention everything. That is not a screen for capability; it is a screen for CV formatting.

The right answer is that unprovable requirements are excluded from the calculation entirely and surfaced as questions for the screening call.

### 4. Does the system ever reject anybody?

It should not. Rank, flag, sort, band — yes. Automatically move a human being out of a process with no person involved — no. Partly because it is the kind of automated decision-making that attracts specific legal obligations in several jurisdictions, and partly because the candidate worth placing is disproportionately the one your rules did not anticipate.

## On what it remembers

### 5. Is an evaluation a new row, or an update?

When a candidate is re-scored — because the rubric changed, or because their profile was updated — does the system write a new evaluation, or overwrite the old one?

If it overwrites, then the score that was in front of the recruiter on 4 March no longer exists anywhere, and the reason a candidate was rejected has been silently replaced by a reason nobody acted on. Append-only evaluation history is a small engineering decision that is nearly impossible to retrofit once there is a year of data, which is why it is worth asking before you buy rather than after.

### 6. Are the criteria versioned, and does each evaluation point at a version?

Related but not the same. The bar moves during a search. If your rubric is a single mutable object, then changing it in week three changes the apparent basis of every decision made in weeks one and two — retrospectively, and with no record that anything happened.

### 7. What does it do when the model is unavailable?

API outages happen, keys expire, and rate limits arrive at the worst moment. There are three possible behaviours: fall back to the deterministic score and say so on screen, refuse and say so, or quietly return something worse without mentioning it. Only the third is disqualifying, and it is more common than you would like.

### 8. Does a status change survive a later status change?

If a candidate reaches final interview and is then rejected, does the record still show that they reached final interview? A system that stores only a current status has thrown away the history you would need to answer almost any question about fairness.

## On who can see what

### 9. Is access enforced by the database or by the page?

Scores, salary expectations and notes are sensitive. Ask specifically whether the restriction is in the data layer or in the user interface — that is, whether someone calling the API directly gets the same refusal as someone clicking the button. Interface-only permissions are a lock on the door of a room with no walls.

### 10. What happens to the CVs of people you did not hire?

A retention period, an enforcement mechanism, and a way to answer a deletion request without a week of searching. If the answer is “they stay in the database”, you are buying a growing liability with a search box on it.

### 11. Is our data used to train models?

Ask about the vendor’s own training and about their subprocessors’. Get the answer in the contract rather than on a call, because the answer on the call is always no and the answer in the terms is occasionally different.

## The regulatory backdrop

Automated tools used to screen candidates are increasingly regulated, and the obligations generally land on the employer or the agency using the tool rather than on the company that built it. A vendor saying “we are compliant” is answering a question about themselves.

- **New York City Local Law 144.** Automated employment decision tools used to screen candidates for jobs in NYC require an annual independent bias audit, publication of a summary of the results, and notice to candidates before the tool is used. Practically, this means asking a vendor what data they will hand over so an auditor can do the work.
- **The EU AI Act.** AI used in employment and worker management is classified as high-risk under Annex III, which brings obligations around risk management, data governance, logging, human oversight and transparency. The compliance timetable has been adjusted more than once since the Act entered into force, so check the dates that apply to you rather than relying on a summary — including this one.
- **Illinois.** The Artificial Intelligence Video Interview Act sets notice, explanation and consent requirements for AI analysis of video interviews, along with deletion obligations on request. Separate amendments to the Illinois Human Rights Act address AI use in employment decisions more broadly.
- **Data protection law generally.** Under the GDPR and UK GDPR, individuals have rights concerning decisions based solely on automated processing that produce legal or similarly significant effects, plus the usual rights of access and erasure over everything you hold about them.

This is a description of the landscape, not legal advice, and it is a snapshot written by a software company rather than a lawyer. Which jurisdictions apply depends on where the role is, where the candidate is and where you are — often all three. If you place people in more than one country, this is worth an hour of somebody qualified.

## What the answers add up to

Notice that nine of the eleven questions are about records rather than about intelligence. That is the point. The quality of the ranking is what you evaluate in a trial, with your own roles, in a fortnight. The quality of the record-keeping is what you cannot evaluate later, because by the time you need it, the year of data has already been written in whatever shape the vendor chose.

If you ask only one, ask the third: what happens to a requirement the CV cannot prove. The answer tells you whether anyone at the vendor has thought about the candidate’s side of this at all.

In Echo Recruit

Our answers to our own eleven

Scoring is deterministic first: skills weighted 50%, experience 25%, role match 25%, against the requisition’s own brief, with the weights on screen. Claude is used for three things — reading a CV, drafting a rubric, and ranking an already-scored slate with its reasoning shown — and is told the numeric score is evidence rather than verdict. Turn the key off and the deterministic path still runs and labels itself.

Rules a profile cannot prove are excluded from the arithmetic and become screening questions. Evaluations are append-only and rubrics are versioned, so each decision points at the rules that produced it. Status transitions are recorded, so a rejection never erases that somebody reached final interview. Nothing is ever rejected automatically. Commercial fields are restricted by row-level security in the database rather than by the page. Customer data is not used to train models.

Share [facebook-f icon](http://www.facebook.com/share.php?u=https://www.echorecruit.com/blog/eleven-questions-to-ask-before-you-let-software-rank-candidates) [linkedin-in icon](http://www.linkedin.com/shareArticle?mini=true&url=https://www.echorecruit.com/blog/eleven-questions-to-ask-before-you-let-software-rank-candidates) [twitter icon](https://twitter.com/intent/tweet?url=https://www.echorecruit.com/blog/eleven-questions-to-ask-before-you-let-software-rank-candidates) [pinterest-p icon](http://pinterest.com/pin/create/link/?url=https://www.echorecruit.com/blog/eleven-questions-to-ask-before-you-let-software-rank-candidates) [envelope icon](mailto:?body=https://www.echorecruit.com/blog/eleven-questions-to-ask-before-you-let-software-rank-candidates)

linkedin-in icon

 

Copyright © 2025, Echo Recruit

```json
{
  "@context" : "https://schema.org",
  "@type" : "BlogPosting",
  "author" : {
    "@type" : "Person",
    "name" : "Echo Recruit",
    "url" : "https://www.echorecruit.com/blog/author/echo-recruit"
  },
  "dateModified" : "2026-09-29T05:39:41.550Z",
  "datePublished" : "2026-09-29T05:39:41.000Z",
  "headline" : "Eleven questions to ask before you let software rank candidates",
  "mainEntityOfPage" : {
    "@id" : "https://www.echorecruit.com/blog/eleven-questions-to-ask-before-you-let-software-rank-candidates",
    "@type" : "WebPage"
  },
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://www.echorecruit.com/hubfs/image-1789710173979.png"
    },
    "name" : "Echo Recruit"
  }
}
```

```json
{
  "@context" : "http://schema.org",
  "@type" : "Article",
  "author" : {
    "@type" : "Person",
    "name" : [ "Echo Recruit" ]
  },
  "datePublished" : "2026-09-29T05:39:41+0000",
  "description" : "Every vendor says their screening is fair and explainable. These are the eleven questions whose answers you can verify — and the one that says walk away.",
  "headline" : "Eleven questions to ask before you let software rank candidates",
  "image" : "",
  "publisher" : {
    "@type" : "Organization",
    "logo" : {
      "@type" : "ImageObject",
      "url" : "https://www.echorecruit.com/hubfs/image-1789710173979.png"
    },
    "name" : "Echo Recruit"
  },
  "url" : "https://www.echorecruit.com/blog/eleven-questions-to-ask-before-you-let-software-rank-candidates"
}
```