9.9.2 Face Detection and Face Recognition

Detecting human faces and recognizing faces and facial expressions have always been an area of interest for different applications such as games, utilities and even security. With the advancement of machine learning, the techniques of detection and recognition have become more accurate and precise than ever before.

However, machine learning remains a relatively complex field that could feel intimidating or inaccessible to many of us. Luckily, in the last couple of years, several organizations and open source communities have been developing tools and libraries that help abstract the complex mathematical algorithms in order to encourage developers to easily create learning models and train them using any programming languages.

Artificial Intelligence (AI) and Machine Learning in particular don't have to be difficult and we hope that the FaceDetect framework gives developers the means to include face detection and recognition in a seamless way in their applications.

The Caligrafy FaceDetect framework was built on top of a FaceAPI library developed by Vincent Muhler and relies on TensorFlow models. FaceDetect can be used directly on the Web without the need to directly manipulate TensorFlow models. It uses pre-trained models for face detection and recognition.

Learn more about the FaceAPI Library here

FaceDetect Framework

FaceDetect is a client-side framework. All the logic happens on the browser side using Javascript. Therefore, just like creating any client-side application in Caligrafy, we only need to create the route on server-side (in the application folder) to render a client-side view.

Refer to the Caligrafy and Vue documentation to know more about how to route to client-side views

The FaceDetect framework relies on 5 main components:

• FaceDetector Class: This is the core class for using FaceDetect in your application. The core code is in detect.js that can be found in the public/js/services folder.

• neural network models: Every machine learning application relies on trained models to be able to do things such as object detection or recognition. FaceDetect is no exception to that rule and it needs these models to operate properly. These models are included in Caligrafy and can be found in public/resources/models.

• recognition models: In order to recognize specific people, FaceDetect needs models to compare the detections to. These models are nothing more but a series of pictures of people organized in folders. These need to be created in your application folder under a folder called recognition. As an example, the facedetect app that comes with Caligrafy has a recognition folder. Inside that folder, you can find a subfolder called Flash that has 6 (PNG) pictures named by number of the superhero Flash. The same mechanism should be used to create more models.

• your app: Your app can be a Vue app or any other client-side app that uses javascript to consume the FaceDetector class. In Caligrafy, all apps need a folder in the public folder with a scripts subfolder and an app index file.

• faceapi.js: The FaceAPI developed by Vincent Muhler is included in the package.

Learn more about the app structure in the Caligrafy and Vue integration section.

Understanding FaceDetect

In order to illustrate the different features of FaceDetector, Caligrafy comes prepackaged with an application that uses some of its features. The documentation will rely on it for illustrating the different concepts.

Face Detection VS Face Recognition

Detection and Recognition are 2 different concepts. While both use machine learning and neural networks in particular, they achieve different things.

• Detection: Detection is about identifying a human face among all other "things" perceived through either an image or a video. So for example, a picture or a video can have people, objects, scenery etc... Face detection is when the system is capable of pointing the presence of a human face among all those other things.

• Recognition: Recognition is about identifying who the human face is among all other faces and things perceived. So for example, Face recognition is when the system is capable of pointing out "Flash" among all other superheroes in a picture or a video.

Understanding the distinction between detection and recognition is key to understanding the underlying logic of the FaceDetector class.

FaceDetector logic

FaceDetector relies on an important principle: "First you detect then you do something with the detections". With that principle in mind, the framework focuses on providing an easy-to-code sandbox for you to do something with the detections. Each sandbox is an application of its own. So if in your application you intend to detect the age of a face or detect the facial expressions or count the number of people or recognize a face etc.. Each one of those is referred to as an application.

FaceDetect Markup Components

Preparing the markup

Your application index.php file is the user interface that will define the source of the face detections (image or video) and any other controls needed to make something useful with the detections. You will need to include the FaceDetect needed libraries into the markup with and without Vue.

• Without Vue

<!-- Initialization scripts -->

<script src="<?php echo APP_SERVICE_ROOT.'app.js'; ?>"></script>
<script src="<?php echo APP_SERVICE_ROOT.'face-api.min.js'; ?>"></script>
<script src="<?php echo APP_SERVICE_ROOT.'detect.js'; ?>"></script>
<script>loadEnvironment(`<?php echo $env; ?>`);</script>

<!-- Additional scripts go here -->
<script src="<?php echo scripts('bootstrap_jquery'); ?>"></script>
<script src="<?php echo scripts('bootstrap_script'); ?>"></script>

<!--[if lt IE 9] -->
<script src="<?php echo scripts('fallback_html5shiv'); ?>"></script>
<script src="<?php echo scripts('fallback_respond'); ?>"></script>
<!--<![endif]-->

• With Vue

<!-- Initialization scripts -->
<script src="https://unpkg.com/vue@3"></script>
<script src="https://unpkg.com/axios/dist/axios.min.js"></script>
<script src="<?php echo APP_SERVICE_ROOT.'app.js'; ?>"></script>
<script src="<?php echo APP_SERVICE_ROOT.'face-api.min.js'; ?>"></script>
<script src="<?php echo APP_SERVICE_ROOT.'detect.js'; ?>"></script>
<script>loadEnvironment(`<?php echo $env; ?>`);</script>
<script> 
   /* Loading the app client framework
    * Any environment variables to be passed on from the server can take place in this here
    */
   loadVue({
      scripts: ['main']
    });
</script>
        
<!-- Additional scripts go here -->
<script src="<?php echo scripts('bootstrap_jquery'); ?>"></script>
<script src="<?php echo scripts('bootstrap_script'); ?>"></script>
        
<!--[if lt IE 9] -->
<script src="<?php echo scripts('fallback_html5shiv'); ?>"></script>
<script src="<?php echo scripts('fallback_respond'); ?>"></script>
<!--<![endif]-->

FaceDetect id

FaceDetect is identified in the markup by the id detector. FaceDetect will be encapsulated within that block. If you are using Vue, the detector block lives within the Vue app id.

<!-- Vue app -->
<div id="app">
   <section id="detector">
   ....
   </section>
</div>

Media Source

No matter what you want to do with FaceDetect, detection is the first step to it. It is therefore important to identify what the detection source is. Is it an image, a video or a live webcam?

The markup of your application needs to provide that source:

<!-- Beginning of the app -->
<div id="app">
   
   <section id="detector">

      <!-- media can be an image 
      <img id="detection" class="show" src="">
      -->

      <!-- media can be a video 
      <video id="detection" src="<video local url>" type="video/mp4" width="720" height="560" class="show" autoplay="autoplay" muted playsinline controls></video>
      -->

      <!-- media can be live webcam -->
      <video id="detection" width="720" height="560" class="show" autoplay="autoplay" muted playsinline></video>

   </section>

</div>

Controls

So far, only the source has been specified. In order to do something with it, it needs one or more UI triggers to activate it. FaceDetector provides you with a way to create these controls if you desire. In order to do that, you will need to add the controls placeholder to your markup.

<section class="controls">
   <div id="apps"></div>
</section>

Infobar

FaceDetect provides you with a UI component to display welcome messages, status or instruction messages called Infobar. In order to use it, you will need to add the infobar placeholder to your markup.

<section id="infobar"></section>

CSS

A basic stylesheet is provided in the facedetect example app to illustrate how the media sources, controls and infobar can be styled if you decide to use them.

FaceDetector class

If you are using the Vue integration, you will need to indicate in the loadVue script described in the markup the name of the javascript main Vue file. In the example provided, we call it main.

If you are not using Vue, link a javascript file called main.js to the markup. We will refer to this file as main for convenience. It could have any name.

Whether you are using Vue or not, FaceDetect has not been written for VueJS. In fact we will integrate it in a VueJS structure but it is still used as an independent JS class. The code examples that we will show moving forward use VueJS but the same can be done using standard JS.

Initializing FaceDetector

FaceDetector is an object that can be initialized by providing it the HTML ID of the media source. It is therefore important to make sure that the markup for either an image or a video has an id attribute.

Upon successful instantiation, all the models are loaded and the features of the class can be used.

// make sure to use the HTML id of the media source
var detector = new FaceDetector('detection');

var app = Vue.createApp({
  el: '#app',
  data () {
    return {
        detector: detector, /* important */
        env: env

    }
  },
  /* Method Definition  */
  methods: {
      
      
  },
  /* upon object load, the following will be executed */
  mounted () {
      
      
  }

});

// mount the app
app.mount('#app');

Running FaceDetector

FaceDetector provides you with 4 out-of-the-box sandboxes that you can use to benefit from face detection and face recognitions.

Upon instantiating a sandbox several things may happen:

1. A button is created that allows triggering/starting the detection
2. A canvas is created overlapping the media source, allowing you to draw on top of the image or video
3. If you are using the infobar, it is instantiated with a message that you specify

Basic detection

The basic sandbox is a detection sandbox that will draw a box around the detected faces in the media source. In order to run the basic sandbox, the loadApp method is called on the instantiated FaceDetector object.

<!-- without Vue -->
detector.loadApp();

<!-- with Vue -->
this.detector.loadApp();

Draw

A more elaborate sandbox is one where you can configure what to draw on the detections. It offers several options that are additive when present.

<!-- Vue example -->
this.detector.loadApp({
    name: 'Name of the app',  // name that will appear on the button created
    method: this.detector.draw, // or detector.draw without Vue
    options: {
       welcome: "The message to display in the infobar",
       detection: true, // draws a box around the detected faces when set to true
       landmarks: true, // line draws the detect faces when set to true
       gender: true, // detects the gender - male or female - when set to true
       age: true, // estimates the age when set to true
       expression: true // detects happy, sad, angry or surprised when set to true
    },
    algorithm: faceapi.SsdMobilenetv1Options // different algorithms can be used for detection. The default algorithm is faceapi.TinyFaceDetectorOptions

});

Recognize

The recognition sandbox is another one that recognizes the detections. If you would like to run recognitions, there are 2 steps that you need to do:

Step 1: Define the recognition models

The easiest way to define a recognition model is to create a recognition folder in your app main folder. The recognition folder will contain many pictures of people organized by person name. Each person to be recognized will have a folder in their name and the pictures inside must be PNGs and must be named with a number.

So for example, if you are detecting the Flash, in the public/<your app folder>/recognition/Flash, you will have a list of of PNG files such as 1.png, 2.png etc.

The same number of images need to be provided for all the models. The lack of doing that may result in unexpected behaviors.

Step 2: Run recognition

Running the recognition is as easy as loading a detection app.

<!-- Vue example -->
this.detector.loadApp({
    name: "Recognize",
    method: this.detector.recognize,
    models: {
         labels: ['Flash', 'Person X', etc...] // array of all the names of the people which are also the names of the folders in the structure
         sampleSize: 6 // number of pictures per person (this number must be the same for all)
    },
    options: {
        welcome: "The message that you want to display in the infobar",
        recognition: true // the recognition engine needs to be activate
    },
    algorithm: faceapi.SsdMobilenetv1Options // Optional. The detection algorithm that will be used
})

Custom

The 4th sandbox is a custom sandbox. For more flexibility, the FaceDetect framework provides you with the ability to invoke your custom methods at specific points of the face detections.

• Continuously, during face detection

If you want to do something with the detections or the recognitions continuously while a video or webcam are continuously detecting and/or recognizing faces, you need to invoke your own method from the loadApp and to set the custom setting to false.

this.detector.loadApp({
  name: "Custom continuous",
  method: this.continuousMethod,
  custom: false, // set to false if you want the method to be applied continuously at every interval of detection
  options: {
      welcome: "Open the console to see how it is continuously being called at every detection",
      detection: true
  }
});

The continuousMethod() needs to be defined as a your own method in Vue or in JS. It takes the object facedetector as an argument.

// Vue example
var detector = new FaceDetector('detection');

var app = Vue.createApp({
  el: '#app',
  data () {
    return {
        detector: detector,
        env: env

    }
  },
  /* Method Definition  */
  methods: {
      
      continuousMethod: function(facedetector) {
          // you can access detections by calling the property facedetector.app.detections. It returns an array of all faces detected every 100ms
          console.log(facedetector.app.detections);

         // you can access recognitions by calling the property facedetector.app.recognitions. It returns an array of the recognized faces every 100ms. recognitions is only available after the recognize method is called
         //console.log(facedetector.app.recognitions;
      }
      
  },
  ...

• Or, through a callback method that will allow you to manually control face detection and recognition

If you would like to have full control over the framework, you can load a custom app by setting the custom property to true and invoking a callback method in the method property.

this.detector.loadApp({
          name: "Custom callback",
          method: this.callbackMethod,
          custom: true, // set to true if you want the method to do something else before calling in FaceDetect features
          options: {
              welcome: "Open the console to see how it is executing its content and waiting for more to be done",
              detection: true
          }
          
});

When the callbackMethod() is invoked, only the app is initialized with the canvas initiated but no detection nor recognition is started. You can control all the detections, recognitions, and UI elements manually.

// Vue example
var detector = new FaceDetector('detection');

var app = Vue.createApp({
  el: '#app',
  data () {
    return {
        detector: detector,
        env: env

    }
  },
  /* Method Definition  */
  methods: {
      
      callbackMethod: function(facedetector) {
         
        //<---- do whatever you want here
          console.log('hello');
        
        /* use any of the FaceDetect methods
         * facedetector.loadApp(app): load another app
         * (facedetector.detectFaces(app, facedetector))(): self invoking function to start face detection
         * facedetector.detect(callback, recognize = false, fetchRate = 100): starts a parallel stream that captures any detections or recognitions when available
         * facedetector.prepareCanva(options = null): returns a new canvas on top of the media source
         * facedetector.draw(facedetector): draws the detections on the canvas
         * facedetector.loadRecognition({ labels: [], images: [], sampleSize: 100}): load models to recognize by the recognition engine
         * facedetector.recognize(facedetector): runs the recognition engine and draws on canvas. Must make sure that detections is started before
         * facedetector.fetchImage(canvas, media): takes a canvas capture of the media and returns a blob data image (data url)
         * facedetector.display(message, output): displays a message in the infobar and gives it an ID as specified by the 'output' input
         * facedetector.clearDisplay(): clears the infobar display
         */
      }
      
  },
  ...