Introduction

As annotation projects become increasingly complex, maintaining consistent quality requires more than traditional review processes.

Many projects now involve multiple object classes, detailed annotation guidelines, and complex relationships between annotated objects. Ensuring that these requirements are applied consistently requires a combination of human expertise and, whenever possible, automated validation.

At People for AI, automated quality checks are integrated into our overall annotation methodology and quality control process that combines real-time quality monitoring, sampling-based reviews, and project-specific validation rules. Together, these complementary approaches help detect quality issues early, maintain consistency throughout production, and continuously improve annotation quality.

In this article, we focus on one component of this methodology: automated quality checks. We explain how they complement human review through two approaches: statistical checks, which identify unusual patterns and potential anomalies, and rule-based checks, which verify compliance with predefined project requirements.

Finally, we illustrate this approach with a real-world case study showing how project-specific validation rules can significantly reduce manual review effort while improving annotation consistency.

From manual quality review to automated quality checks

Quality control is a critical part of any annotation project. However, reviewing every single annotation manually is often not scalable when working with large datasets.

To maintain quality throughout our annotation workflows, we have implemented a multi-level quality control approach combining real-time monitoring, sampling-based reviews, and automated checks. 

Real-time quality monitoring

During the annotation process, our teams regularly review recently completed annotations and provide feedback directly to annotators.

The objective is to detect potential deviations from the expected quality standards as early as possible and prevent the same issue from spreading across a larger volume of data.

For example, in a dermatology annotation project, our quality team noticed that one annotator had started creating lesion masks that were significantly smaller than expected. This deviation was quickly identified through regular monitoring, allowing us to provide feedback and correct the annotation approach before it impacted a larger batch of data.

This type of review is essential because it allows continuous improvement and alignment with project requirements.

Sampling-based quality review

In addition to real-time monitoring, quality teams typically perform reviews on a representative sample of completed annotations.

For example, a defined percentage of the production output can be reviewed to assess whether quality standards are consistently maintained.

This approach provides a balance between quality assurance and operational efficiency. Reviewing 100% of annotations manually is possible only when explicitly planned and agreed upon with the client, as it requires significantly more resources.

Where automation adds value

Some quality checks are repetitive, objective, and based on clearly defined rules. For these checks, manual review does not necessarily provide additional value.

Automating these validations allows quality teams to focus their time and expertise on cases that require human judgment, while ensuring that systematic checks are applied consistently across the entire dataset.

Examples include:

• Size and surface checks,
• Spatial relationship checks between objects,
• Object presence and cardinality checks,
• Detection of annotation artifacts and noise.

Automated checks therefore complement existing quality processes by adding an additional layer of validation after annotation completion.

Two complementary approaches to automated quality checks

Automated quality checks can be implemented in different ways depending on the type of issue we are trying to detect.

In our experience, two complementary approaches are particularly effective.

Statistical checks

Statistical checks aim to identify annotations that behave differently from the rest of the dataset.

Rather than relying on predefined rules, they look for unusual patterns, outliers, or inconsistencies across a set of annotations. These analyses can be performed at image level, object level, or even annotator level.

Examples include:

• Masks that are significantly smaller or larger than the rest of the dataset,
• Objects whose position differs from similar annotations,

Statistical checks are particularly useful because they can reveal issues that were not anticipated when the annotation guidelines were defined.

For example, by analyzing the distribution of object sizes across a dataset, it is possible to identify a small number of annotations that differ significantly from the rest. While these cases are not necessarily incorrect, they often deserve closer inspection and can help uncover annotation inconsistencies, edge cases, or misunderstandings of the guidelines.

Rule-based checks

Rule-based checks verify that annotations comply with explicit project requirements.

These requirements are typically defined by the client, described in the annotation guidelines, or derived from known relationships between annotated objects.

Examples include:

• Verifying that one object is contained within another,
• Ensuring that two regions do not overlap or do not touch each other,
• Checking that expected relationships between objects are respected,
• Detecting segmentation artifacts or annotation elements that should be excluded from the final dataset.

Because these checks rely on objective and repeatable rules, they can be applied systematically across large volumes of annotations and generate targeted correction tasks for QA teams.

However, not every possible validation rule should necessarily be automated. Designing an automated check requires finding the right balance between implementation effort, reliability, and expected impact. 

One of the challenges when designing automated quality checks is determining where to start and where to stop: a complex check may require significant development and maintenance efforts while only detecting a limited number of issues. Conversely, a simpler validation rule that can be applied consistently across the entire dataset may provide much greater value by reducing manual review effort and preventing recurring errors.

In practice, the most valuable checks are not necessarily the most complex ones, but the ones that significantly reduce manual review effort while improving overall dataset quality.

Combining both approaches

Although statistical and rule-based checks are conceptually different, they are often complementary in practice.

A statistical analysis may highlight unusual annotations that eventually turn out to be violations of project requirements. Conversely, rule-based checks can provide a precise explanation for anomalies initially identified through statistical methods.

The objective is not to automatically determine whether an annotation is correct in every situation. Instead, these checks help identify suspicious cases, generate actionable review reports, and provide quality metrics that support continuous improvement of the annotation process.

By combining statistical analyses, project-specific validation rules, and human expertise, quality teams can focus their efforts where they create the most value while maintaining consistent quality standards across the dataset.

Case study 1: automated quality checks for hockey rink annotations

The objective of the project was to annotate the different areas of a hockey rink. Each area was geometrically defined in the annotation guidelines, leaving no room for interpretation. This made the project a good candidate for automated validation, as the expected relationships between annotated regions could be translated into explicit rules. 

Based on these project requirements, we implemented an automated check called IMP_TOUCH to identify cases where annotated elements were in contact despite not being expected to touch.

The first step was to define the valid spatial relationships between the different classes and establish the list of class pairs that were allowed to be in contact.

The output is a list of images requiring review, allowing QA teams to focus directly on the potentially problematic cases rather than manually inspecting the entire dataset.

Defining the right validation scope

Based on the annotation guidelines, we could define that a face-off circle should always contain an outside red face-off dot. However, some face-off circles were partially visible or cropped in the image, making it impossible to determine whether the expected relationship was actually respected.

Applying this rule in all cases would have generated a high number of false positives. To keep the check reliable, we decided not to apply the validation when the face-off circle was partially outside the image boundaries.

On the other hand, when the complete object was visible, we were able to safely enforce the rule that the outside red face-off dot had to be contained within the corresponding face-off circle.

While it would have been possible to further refine the validation to handle additional edge cases, the expected benefit did not justify the additional implementation complexity. The chosen approach provided a good balance between reliability, development effort, and the number of potential issues detected. 

Case study 2: temporal consistency checks for hockey player annotations

In a second hockey annotation project, our client provided a plotting version of our annotations, allowing us to visualize player positions and attributes using the tracked coordinates.

While this visualization was useful for reviewing annotation quality, we quickly realized that manually verifying every player across every frame of an entire match was not realistic. With multiple players, identities, team assignments, and jersey numbers to validate simultaneously, many inconsistencies could easily go unnoticed.

To support our QA process, we developed a suite of automated validation rules designed to verify the consistency of player tracking and jersey identification across entire game sequences.

Unlike traditional image-level checks, these validations combined temporal, spatial, and domain-specific rules to identify annotation errors that would be difficult to detect through manual review alone.

The automated checks covered several categories:

• Identity consistency, such as detecting cases where the same player ID was associated with different jersey numbers or switched teams during a sequence.
• Missing information, including empty jersey numbers or team assignments.
• Game-rule validation, such as detecting duplicate jersey numbers within the same team, multiple goals annotated on the same frame, or an unrealistic number of players on the ice.
• Temporal anomaly detection, identifying players who briefly appeared with a different jersey number before returning to their original identity, often indicating a tracking or annotation error.
• Behavioral and spatial checks, including players remaining almost stationary near the rink boundaries for long periods, which often revealed substitutes, coaches, or referees incorrectly annotated as active players.
• Goalie consistency checks, ensuring that goalies maintained the same team assignment throughout a period.
• Occlusion-related inconsistencies, where a player disappeared behind another player and later reappeared with a different jersey number or identity.

To reduce false positives, most temporal checks were only reported when the anomaly persisted for at least 30 consecutive frames.

Rather than manually reviewing an entire match, annotators could focus directly on the sequences flagged by the validation engine. In practice, the automated checks became a true annotation assistance tool, reducing review and correction time from up to 15 hours per match to approximately 2–3 hours. 

Example of checks performed by an annotator:

Conclusion

Quality in annotation does not rely on a single method, but on the complementarity of several layers of control: real-time monitoring to catch deviations as soon as they appear, sampling-based reviews to ensure overall consistency, and automated checks to apply objective rules at scale.

These checks do not replace human expertise, they amplify its impact. By automatically filtering suspicious cases across large volumes of data, they allow QA teams to focus their efforts where they matter most: on the ambiguous situations that require human judgment. This is the approach that allowed us, for example, to reduce match review time from up to 15 hours to approximately 2–3 hours without compromising quality.

Of course, not every project lends itself to automated validation. Effective checks require rules that can be applied reliably, sufficient expected value, and a clear benefit compared to their implementation effort. But when these conditions are met, automated checks can become much more than a final layer of verification: they can evolve into a true annotation assistance tool, helping annotators and reviewers work faster, more consistently, and at greater scale.

Working on an annotation project with complex quality requirements? Let’s talk about how we can build automated checks tailored to your data.