I will be upfront about something. I do not like managing people. Never have. When I am managing, I get annoyed at inconsistencies. I try to do things perfectly and when others do not, I get difficult to work with. So I figured out a long time ago that the best thing I can do is stay out of it and build systems that manage the work instead.
That is genuinely how the automated quality control pipeline for our data annotation projects came to be. It was not born from a grand vision about efficiency or scalability, though those things are real benefits. It was born from a practical realisation: we had hundreds of contributors submitting data through our collection portals every day, all of that data needed to be quality checked against specific criteria before it reached the client, and the idea of a team of humans reviewing it file by file was something I found both expensive and unreliable. If the review was going to be done consistently, it needed to be done by something that did not get tired, bored, or inconsistent on a Wednesday afternoon.
So I built it. Here is how it works.
An Honest Reason to Build Automation
When I joined ConsultBae, the AI data vertical was not really a vertical. We were doing one project, borrowing interns from the recruitment team when we needed people, and sending them back when the project was done. As the work grew and we formalised the vertical, it became clear that quality control was going to be the thing that either made the operation credible or killed it.
The developer in me had been dormant since I left my Android app days and moved into sales. Working here, it woke back up. Last year I rebuilt the whole website from scratch, then built the image collection portal, then the email collection portal, then started getting seriously into automation using N8n and Python. Not because someone asked me to. Because the problem was in front of me and building was the obvious answer.
The quality control problem is simple to describe. Hundreds of contributors are submitting files. Every file needs to be checked against a specific set of criteria. If it passes, it gets logged, the contributor's payment record gets updated, and eventually it goes to the client. If it fails, it gets rejected, the contributor gets notified, and the rejection is recorded. That whole chain, at the volume we are operating, is not a job for people. It is a job for a well-designed workflow.
How the Five-Level QC Pipeline Actually Works
All the data comes in through the collection portals and goes into our database. Contributors have their own IDs and their submissions are tagged accordingly. From there, my N8n flow picks it up automatically. N8n is an automation platform that connects to practically any service via API and has AI nodes built in, so you can wire up a database, a drive, and an OpenAI model in the same workflow without writing much code.
Each quality check runs as a separate N8n flow with an OpenAI node at its centre. The node receives the file and a precise set of instructions about what to check. Not vague guidance. Specific, unambiguous criteria: check this, verify that, if this condition is true then pass, if not then reject. The model does not need to interpret. It just needs to apply the instructions consistently, which it does far better than a human reviewer working through a long queue at the end of the day.
I split the checking into five levels because cramming everything into one flow would be expensive, hard to debug, and would give me very little information about where a file actually failed. Five separate levels means the first one catches the obvious problems cheaply, the subsequent ones run on the progressively smaller pool that made it through, and if something is going wrong I can see exactly which level is causing it. It is just cleaner to think about and easier to fix when something breaks.
After each level, the database row for that submission is updated. Pass or fail, and at which stage. That record is not just for auditing. It feeds directly into contributor payments.
"Everything is automated. I just have to keep track of whether it is working correctly or not. Because if it rejects correct assets or passes bad ones, that is on me. The pipeline has to be flawless — because it is also determining who gets paid."
Why the First Filter Rejects 30% Before Anything Else Runs
The first flow rejects around 30% of what comes into the platform. These are the clearest failures: wrong file format, incomplete submissions, things that do not meet the most basic technical requirements of the project. Catching these at the first stage means the rest of the pipeline is only ever running on the 70% that already passed the baseline check. That keeps the whole thing faster and cheaper, because you are not burning compute running five levels of checking on a file that should have been rejected immediately.
By the time a submission has cleared all five stages, it is genuinely ready for the client. From the first flow to the fifth, we typically end up with 55 to 60% of total submissions passing. The other 40 to 45% got caught somewhere in the pipeline. None of it reaches the client. None of it triggers a rejection on their end. The filtering happens on our side, silently, before delivery.
This matters more than it might seem. Every file that gets rejected by a client after delivery creates rework, delays, and a slightly more strained relationship. Every file that gets caught by the automated pipeline before delivery is just a rejected row in our database. Same outcome for the data, completely different experience for the client.
How QC Output Determines Who Gets Paid What
Contributors are paid based on accepted submissions, not total submissions. This means the pipeline's decision on each file is also the payment decision. A file that passes all five levels contributes to that contributor's payout. A file that fails at level two does not.
This creates an obvious requirement: the pipeline has to be accurate. Not just mostly accurate. Flawless, as much as something can be. If it incorrectly rejects good work, I am underpaying contributors who did their job properly. If it incorrectly passes bad work, I am paying for data the client will reject. Both are bad outcomes and both are my responsibility to catch.
So part of my day-to-day is monitoring whether the flows are running correctly. Not reviewing individual files, but watching the aggregate numbers. If the rejection rate at a particular stage spikes unexpectedly, something has changed: either the contributor pool is submitting worse work, or a flow has developed a bug, or the criteria I wrote no longer match the client's current requirements. Any of those is fixable, but only if I catch it early. A problem that runs undetected for three days while hundreds of submissions are being incorrectly assessed is a much worse problem than one caught in the first hour.
The Last Flow: Automated Delivery Straight to the Client's Drive
Once a submission has cleared all five levels, the final flow moves it directly to the client's storage. I have added the client's credentials to the N8n workflow so the upload happens automatically, without anyone on our side packaging or transferring anything manually. The client's drive receives the data on a rolling basis as it passes review, rather than waiting for a single delivery at the end of the project.
The last flow also generates a CSV of the delivered batch in whatever format the client needs: file name, category, subcategory, domain, and any other metadata fields they have specified. This CSV goes to the client alongside the data. They receive clean data plus a structured summary of what they received, all without anyone on our end spending time on delivery administration.
The practical benefit for the client is that they can start working with the data sooner. If there is a systematic problem with how contributors are interpreting the brief, it shows up in the early deliveries rather than at the end of the project when all the data has already been collected on the wrong basis. Early delivery means early feedback, and early feedback is what keeps a large data project from going badly wrong in the final stretch.
The One Thing This System Cannot Do
The pipeline is consistent in a way that a human reviewer is not. It applies the same criteria to submission number one and submission number ten thousand without any degradation. That is its biggest advantage and also its limitation.
There are edge cases that require judgment. A file that technically fails one criteria but contains something genuinely valuable that a human reviewer would recognise as worth passing. A submission that is technically correct but has clearly been produced without genuine effort, something an experienced reviewer would flag but the automated check might miss. The pipeline cannot make those calls. It applies the criteria as written, and if the criteria do not account for the edge case, the edge case gets handled incorrectly.
Managing this means writing the criteria carefully upfront and refining them as the edge cases reveal themselves across projects. Every time the pipeline produces a result that I look at and think that was wrong, I update the relevant flow so it handles that situation correctly next time. Over time, the pipeline gets better because the edge cases it has seen are built into its instructions. It is not a system you build once and leave alone. It is one you continuously improve as the work teaches you what it actually needs.
Level 1 — Technical basics: File format, completeness, and structural requirements. This is the cheapest check to run and the one that filters the most: around 30% of submissions do not make it past here. Running this first means levels two through five are only processing files that already meet the baseline.
Levels 2 through 4 — Progressive content checks: Each level checks a specific set of content quality parameters that the previous level did not cover. The criteria at each stage are written as precise instructions to the OpenAI node, specific enough that the model applies them consistently without judgment calls. The progressively smaller pool at each stage keeps the overall compute cost manageable.
Level 5 — Final compliance pass: The last check before automated delivery. By this stage, the pool is already well-filtered and the final level is confirming full compliance with the project brief before the data moves to the client's drive. After level 5, the delivery flow and the payment CSV generation both run automatically.
Need Data Annotation Services With Quality Infrastructure Built In?
ConsultBae runs automated quality control across all active data collection projects, with five-stage validation, contributor payment tracking, and direct client delivery in the same pipeline. Talk to our team about what your project requires.
Talk to Our AI Data Team
