Most companies that have been burned by an AI data vendor describe the same arc. The initial conversations went well. The proposal looked thorough. The first delivery arrived on time. Then something went wrong on a later project — quality dropped, edge cases were handled poorly, a batch failed review and nobody on the vendor side seemed equipped to fix it quickly — and in the post-mortem, the buyer realised they had never asked the questions that would have surfaced the actual capability of the vendor before they signed.
The conversations that should happen before a data contract is signed are not the ones most procurement processes are designed to have. Standard vendor evaluation focuses on price, volume capacity, and reference clients. Those matter. They are also not predictive of whether the vendor can actually run the work well. Five other questions are, and they consistently separate vendors who operate from vendors who resell capacity.
Question one: how do you source contributors and verify their identity
Where the data actually comes from is the most foundational question and the one most buyers do not ask in any specific detail. The answer reveals whether the vendor has direct relationships with its contributor network or is layering on top of crowdsourcing platforms that they themselves do not control.
A vendor with direct relationships can answer specific questions: how contributors are recruited, what verification they go through, what demographic data is collected, how contributors are matched to projects, what the vendor does when a contributor's quality drops. A vendor that resells capacity from third-party platforms will give general answers because the specific details are not theirs to know.
The identity verification question is the second part of this. For specialist projects, the vendor needs to be able to confirm that the contributor actually has the demographic profile, language, or domain background the project requires. If they cannot describe how they do that, the project will end up with data that does not match its specification, and the buyer will discover this in production rather than during collection.
Question two: what is your annotation quality process and can you show inter-annotator agreement data
Every vendor will claim a strong quality process. The useful version of this question is the request for evidence. Inter-annotator agreement is the metric that actually shows whether multiple annotators working on the same task produce consistent labels, and it is a metric that mature annotation operations track routinely.
A vendor that can produce inter-annotator agreement data from previous projects, ideally broken down by task type and annotator experience level, is a vendor whose quality claims are based on measurement. A vendor that talks about quality in general terms without being able to show the underlying metrics is making claims that may or may not be true and that the buyer has no way to verify.
A vendor that cannot show you inter-annotator agreement data is telling you they have not been measuring it. Whatever quality claim they make is anecdotal at best.
Question three: how do you handle edge cases that fall outside the brief
Every meaningful data project encounters cases that the original brief did not anticipate. How the vendor responds to those cases is one of the highest-leverage operational details in the entire engagement, and it is rarely discussed before a contract is signed.
The right answer involves a defined escalation process: contributors and first-line reviewers can flag cases that do not fit clearly, those cases route to a specific role within the vendor team, and decisions about how to handle them are documented and shared with the client so the brief can be updated. The wrong answer is some version of contributors resolving edge cases with their best judgment, which produces datasets with quietly inconsistent handling of exactly the cases where consistency matters most.
Question four: what is the chain of custody for the data from collection to delivery
Chain of custody covers everything that happens to the data between contributor and client. Who touched the file, what processing was applied, where it was stored, what compliance and security controls were in place, who reviewed the quality and at what stage.
A vendor with operational maturity can walk through this in specific detail. A vendor without it will answer in general terms about security and quality without being able to describe the actual workflow the data moves through. For sensitive use cases — anything involving personal data, health information, or commercial intellectual property — the chain of custody is not a procurement formality. It is the operational reality that determines whether the data the client receives is what they were promised and whether the project carries compliance risk the client did not know it was carrying.
Question five: what happens if a delivery batch fails quality review
The most revealing question to ask any vendor is what they do when something goes wrong. A vendor that has done enough work to have failed deliveries in their history will answer this question concretely, because they have a process built from learning what does and does not work when a batch needs to be reworked or re-collected.
A vendor that has not done enough work, or has not been paying attention to what does and does not work in the rare cases of failure, will answer in reassurances rather than specifics. "We have very high quality standards" is not an answer to this question. "Here is how we identify the root cause, here is who owns rework, here is our typical rework timeline and where we absorb cost versus where we ask the client to absorb it" is an answer.
Vendors who can answer these five questions in specific operational detail are vendors who run the work themselves. They have a contributor network they understand, a quality process they measure, edge case workflows they have refined through experience, and operational recovery procedures built from real failures.
Vendors who answer in general assurances are reselling capacity. They may still deliver a successful project. They may also be the source of the problem the buyer discovers in production six months later. The cost of asking these questions before signing is zero. The cost of not asking can be the entire project.
How ConsultBae approaches this
We invite these questions early in client conversations because we have answers to all five and the answers are how we differentiate from the part of the market that does not. Our contributor network across 100-plus countries is directly managed. Our quality processes are measured and the data is available for review. Our edge case escalation is documented and operates the same way on every project. Our chain of custody covers collection through delivery with controls appropriate to the sensitivity of the work. And our recovery process for failed batches has been refined across enough projects that we know how to fix things quickly when they need to be fixed.
The right vendor for an AI data project is not the cheapest one or the largest one. It is the one whose answers to these five questions hold up when the project is six months in and the operational reality is showing through whatever the proposal claimed at the start.
Mohit Singh Katewa works at ConsultBae across operations and delivery for clients building AI, hiring teams, and learning programmes.
Scoping an AI data project?
ConsultBae answers all five questions in specific operational detail. Let us walk you through how we run the work.
Talk to us
