How to Screen AI Candidates: 8 Ways to Verify Real AI Experience

AI experience is appearing on more CVs, but how can hiring managers tell whether a candidate has genuinely developed AI systems or simply used AI-powered tools?

As businesses invest in generative AI, machine learning, and agentic AI, identifying professionals with the right technical capabilities is becoming increasingly challenging. Candidates may reference large language models (LLMs), AI agents, and machine learning frameworks, but these terms alone reveal little about their actual experience.

Charlie Tulio, a technology talent specialist at Glocomms, has encountered this problem firsthand:

The challenge is particularly relevant as AI tools make it easier to produce convincing applications and prepare responses during interviews.

For hiring managers, effective AI candidate screening means looking beyond resume claims to assess practical experience, technical decision-making, and the ability to deliver AI solutions in real business environments. From defining role requirements to verifying project experience and conducting technical assessments, a structured approach can help organizations identify candidates with the capabilities their AI initiatives require.

Here's how to apply these principles throughout the hiring process.

1. Define the AI skills you actually need

Effective AI candidate screening starts before the first interview. One challenge facing employers is the lack of consistency in how AI positions are defined. Job titles such as AI Engineer, Machine Learning Engineer, and Data Scientist are sometimes used interchangeably, despite requiring different technical capabilities.

Charlie highlights this issue in his conversations with clients:

This ambiguity can result in unsuitable applications, inefficient screening processes, and unrealistic expectations about the available talent pool.

Before advertising a position, hiring managers should clarify what the successful candidate will be expected to deliver. For example:

• AI Engineer: Develops AI-powered applications and integrates models into business systems.
• Machine Learning Engineer: Builds, trains, evaluates, and deploys machine learning models.
• Data Scientist: Applies statistical analysis and predictive modeling to business problems.
• MLOps Engineer: Develops infrastructure for deploying, monitoring, and maintaining machine learning systems.
• AI Research Scientist: Investigates and develops new modeling techniques and architectures.

These responsibilities can overlap, particularly within smaller technology teams. The important distinction is whether the organization needs someone to use existing AI models, develop applications around them, or build and train models directly. Clearly defining these requirements helps hiring managers assess candidates against the work they will actually perform.

2. Look beyond AI keywords on a resume

A candidate listing ChatGPT, OpenAI, LangChain, TensorFlow, or PyTorch may have relevant experience, but familiarity with these technologies does not automatically demonstrate technical proficiency. The most useful screening questions explore how candidates have applied AI in practice. Rather than asking whether someone has worked with LLMs, ask them to explain a specific project.

Here are five questions to verify AI experience:

• What AI system did you personally help develop, and what was your contribution?
• Which models, frameworks, and technologies did you select, and why?
• What technical challenges arose during development?
• How did you test the system's accuracy, reliability, or performance?
• What business results did the project deliver?

Experienced candidates should be able to explain their decisions and discuss the limitations of their work. For example, someone claiming to have built an AI-powered customer service application should understand how the system retrieves information, handles inaccurate responses, and protects customer data.

A candidate who can only describe the application's features may have been involved in its use or management rather than its technical development. This does not make their experience irrelevant. It means hiring managers need to establish whether that experience matches the position.

3. Distinguish between AI users, implementers, and builders

One of the most useful ways to evaluate AI candidates is to distinguish between three types of experience.

Experience	What candidates should demonstrate
User	Uses established AI tools to improve workflows, analysis, or productivity.
Implementer	Integrates existing AI models and services into applications or business
Builder	Develops, trains, evaluates, or operates AI models and their supporting systems.

Each category can deliver business value, but they require different skills. A software engineer integrating an existing LLM into a customer-facing application may need strong API development, software architecture, and security expertise. A research engineer developing new model architectures requires deeper knowledge of machine learning, mathematics, and model training.

Charlie has observed particular demand among his clients for professionals with deeper development capabilities:

This reflects the requirements Charlie is seeing within his client base, rather than every organization's AI hiring priorities.

Hiring managers should avoid assuming that building models from scratch is inherently preferable to using existing technologies. The right approach depends on the business problem, available resources, and technical requirements.

The screening process should establish whether candidates have the appropriate level of expertise for the solution the organization intends to develop.

4. Use practical assessments to verify technical ability

A candidate may be able to discuss AI concepts confidently, but can they apply that knowledge to solve real technical problems? Practical assessments allow hiring managers to look beyond interview responses and evaluate how candidates approach challenges they would encounter in the role.

These assessments should reflect the position's responsibilities, focusing on relevant technical skills, problem-solving abilities, and the reasoning behind key decisions.

For AI Engineers

Ask candidates to outline an LLM-powered application, including its architecture, data sources, model selection, and evaluation approach.

For Machine Learning Engineers

Present a modeling problem and ask candidates to explain data preparation, model selection, validation, and performance monitoring.

For Data Scientists

Provide a business scenario requiring predictive analysis and ask candidates to explain their methodology and how they would communicate the findings.

For MLOps Engineers

Ask candidates to describe how they would deploy, monitor, and update a machine learning model in a production environment.

Assessments should also explore technical trade-offs. For example, can the candidate explain when a smaller model might be preferable to a larger one? Can they identify the implications of inference costs, response latency, or data privacy? These discussions can reveal technical judgment that is difficult to assess through conventional interview questions.

Keep exercises time-limited, relevant, and consistent across applicants. Candidates should understand whether AI assistance is permitted and how their work will be evaluated.

5. Test experience beyond AI prototypes

Developing an AI proof of concept is different from operating a reliable system in a commercial environment. Production AI applications require supporting infrastructure, data pipelines, monitoring, and processes for managing performance issues. This is an area where Glocomms is seeing important skills gaps.

Hailey Thompson, a Consultant at Glocomms, explains that employers can struggle to find professionals who combine AI engineering knowledge with infrastructure expertise:

For organizations planning to scale AI applications, this combination of skills can be particularly valuable. Hiring managers should explore whether candidates understand how AI systems operate beyond initial development.

Questions could include:

• How would you monitor an AI model after deployment?
• What would you do if model performance deteriorated?
• How would you manage increasing data volumes?
• How would you control infrastructure and inference costs?
• What steps would you take to protect sensitive information?

Charlie's experience also highlights the importance of infrastructure planning. He has observed organizations concentrating on model development without giving equal consideration to how supporting systems will scale. For employers, this means screening should assess candidates' understanding of the wider technical environment, particularly when hiring for senior engineering or architecture positions.

6. Assess hands-on expertise in senior AI candidates

Seniority does not always translate into the technical capabilities required for an AI leadership position. Some organizations need experienced leaders who can define strategy, manage teams, and communicate with senior stakeholders. Others require leaders who remain directly involved in research, architecture, and implementation.

Hailey has observed growing interest in this second profile, particularly for agentic AI initiatives:

For hiring managers, this creates an important distinction between strategic oversight and practical technical leadership.

A candidate may have extensive experience managing AI teams without having recently developed or deployed an AI system themselves.

When hands-on expertise is required, ask senior candidates to explain:

• Which technical decisions they personally made on recent projects
• How they contributed to architecture, research, or implementation
• Whether they still write, review, or evaluate technical work
• How they approach emerging technologies such as agentic AI
• How they balance delivery responsibilities with longer-term strategy

The objective is not to favor technical involvement over leadership experience. It is to identify the combination the business needs.

For organizations developing new AI capabilities, a leader who can contribute to technical delivery while establishing the team's direction may be particularly valuable.

7. Recognize the risks of AI-assisted interviews

AI tools are creating additional challenges for candidate verification. Generative AI can help applicants prepare resumes, develop interview responses, and complete technical exercises. These uses are not necessarily inappropriate, particularly when employers permit AI-assisted work. However, hiring managers need to establish whether candidates can independently explain and defend the work they present.

Charlie has observed situations where candidates appear technically qualified on paper but struggle to discuss their claimed experience during screening. He also notes that access to ChatGPT during interviews can make it harder to determine whether responses reflect the candidate's own knowledge.

A stronger screening process should therefore include:

• Detailed follow-up questions: Ask candidates to explain why they made particular technical decisions.
• Scenario variations: Change an element of a technical problem and ask how their approach would adapt.
• Live technical discussion: Explore the reasoning behind an assessment rather than relying solely on its final output.
• Consistent assessment policies: Communicate whether AI tools are allowed and apply the same standards to every candidate.

The objective is to verify competence, not to penalize candidates for using tools that are increasingly part of modern software development.

8. Create a consistent AI candidate screening scorecard

A structured scorecard can help hiring managers compare candidates against job-specific requirements.

The following is an illustrative framework for a technical AI engineering position.

Assessment criteria	Suggested weighting
Technical knowledge relevant to the role	25%
Verified project experience	25%
Practical problem-solving	20%
Production and infrastructure knowledge	20%
Communication and business understanding	10%

These weightings should be adapted to the position.

For example, infrastructure expertise may carry greater weight for an MLOps Engineer, while research experience may be more important for a specialist model development position. Documenting evidence against each criterion helps hiring teams make more consistent decisions and reduces reliance on resume terminology or interview confidence.

How Glocomms helps businesses identify AI talent

As AI becomes embedded across business operations, identifying professionals with verified technical experience requires a clear understanding of both the technology and the talent market. At Glocomms, our specialists work with organizations hiring AI Engineers, Machine Learning Engineers, Data Scientists, and other specialist technology professionals.

Our approach combines technical requirement discussions, candidate qualification, and market knowledge to help businesses identify professionals suited to their projects.

We also provide guidance on compensation, talent availability, and hiring requirements, helping organizations develop realistic strategies for securing specialist AI expertise.

Looking to build or expand your AI team?

Contact Glocomms to discuss your requirements and identify professionals with the technical capabilities your business needs.