InstructIQ

This is one of the biggest challenges in our industry and the single greatest barrier to implementing competency-based training.

Measuring competency is hard. It requires that we all agree on what competency means, and what are the behavioural and performance indicators for each competency. The ICAO Document 9995 (Manual of Evidence-Based Training) goes a long way towards defining what these are.

Paladin AI has built up a data bank of performance indicators that goes beyond Doc 9995. We’ve built up this data bank by working closely with highly experienced flight instructors, and by carefully consulting Airman Certification Standards (ACS) and aircraft manuals.

When our system automatically detects these performance indicators, it assesses the pilot’s performance against the indicator in a transparent way. Over the course of a regular training session, a pilot will produce dozens or even hundreds of individual indicators of competency.

We then feed these indicators to our competency inference engine, a sophisticated Bayesian machine learning system that calculates a maximum-likelihood estimate of the competency profile for the pilot during this training exercise.

Instructors are given the final word. While our system is objective in its assessments, only a human in the loop can sign off on the training session.

Full-flight simulators are considered the gold standard for pilot training because of their incredible fidelity to real-life flying. That is why regulators currently insist on their use. In future, objective data collected from training will likely show that much of this training could be shifted to smaller, more affordable devices — even virtual or augmented reality training. But this must first be shown to be equally effective, and PACE’s training analytics will be key in demonstrating this. In the short term, regulators will likely insist on some amount of full-flight simulator training for the first generation of air mobility operators.

The vision of PACE is to replace convention. Training should be personal, based on needs, which differ from person to person based on background and prior experience. The only way to do this in a scaleable way is by leveraging artificial intelligence to better understand those needs and automatically tailor training.

In aviation, “competency” has actually been rigorously defined. Documents such as ICAO’s Manual of Evidence-Based Training (Doc 9995) has an entire appendix devoted to carefully defining each core competency, along with a list of “indicators” that can be directly observed by the instructor to verify whether the competency is present.

Core competencies include: flight path management (in manual and automated modes), situational awareness, application of procedures, communication, leadership and teamwork, problem solving and decision making, and workload management. Some also include “knowledge” or “application of knowledge” as a separate, ninth competency.

Competencies differ from the concept of “proficiency.” Proficiency represents demonstrating a single procedure or maneuver correctly. The core competencies are the underlying qualities possessed by the pilot that are predictive of proficiency. It’s closer to a concept like IQ. And just like IQ, it can only be measured indirectly. But with the right tools, such as artificial intelligence, this indirect measurement is possible.

InstructIQ® is a web and mobile platform for adaptive pilot training.

Flight instructors install the application on their iPads, while PACE supplies the training center with a data collector. Using InstructIQ, flight instructors get access to training analytics, electronic grading sheets, briefing and debriefing tools, and training recommendations.

InstructIQ® enables airlines and training centers (including flight schools) to implement competency-based training at their organization, saving money through more efficient training, while also training directly to the needs of the individual pilot.

In aviation, “human factors” are crucial to consider. Non-technical skills like crew coordination, communication, leadership, managing workload, etc. are just as important as the ability to operate the aircraft and apply procedures.

For the near future, assessing such human factors is still firmly the responsibility of the flight instructor. However, there are some tools on the horizon that could help with assessing non-technical competencies. These are technologies such as eye-tracking, vocal tonality measurements, and other biometrics that can track stress. With these technologies, there is the potential to estimate the pilot’s workload management and communication skills.

Today, Paladin AI is focussed on building automated performance and technical competency assessment tools, but we are envisioning the necessary technologies to make the leap to comprehensive competency assessment.