As part of our recently foray into robotics I’ve been experimenting with facial recognition in Java.

The robot, an Aldebaran Pepper, can recognise and learn faces but only using its camera. This means it’s not currently possible to train Pepper with existing photos. A possible way out of this would be to use third party software. The OpenCV project seems formidable and is very active but unsurprisingly using it is not as simple as downloading a jar file or running an installer.

The facial recognition ability is part of the extra modules. You’ll need to download them here

https://github.com/opencv/opencv_contrib

Along with the latest version of OpenCV.

https://opencv.org/releases.html

Even a slightly out of date copy of the source might not compile. I had problems with 3.3.0 which were fixed when I found 3.3.1.

OpenCV will require CMake and various libraries which will be flagged up as it attempts to configure the compile script. You’ll need to make a build directory in the main OpenCV directory after unzipping then run cmake in there. In the end this worked for me

cmake -D CMAKE_BUILD_TYPE=Release -D CMAKE_INSTALL_PREFIX=/usr/local -D BUILD_SHARED_LIBS=OFF -D BUILD_EXAMPLES=OFF -D BUILD_TESTS=OFF -D BUILD_PERF_TESTS=OFF -D OPENCV_EXTRA_MODULES_PATH=/home/edward/Downloads/opencv_contrib-master/modules/ -D BUILD_opencv_text=OFF ..

The earlier parameters enable a Java build (you’ll need things like Ant too), following that the extra modules parameter points to where you unzipped them. The last text=OFF parameter may not be necessary, it was left over from when I didn’t have 3.3.1 and was switching off modules which were having trouble compiling. As long as “face”, the module we’re after, doesn’t rely on an extra module I see no reason not to switch it off if you’re having compile problems.

CMake output will hopefully include something like this

— To be built: core flann imgproc ml objdetect phase_unwrapping photo plot reg surface_matching video xphoto bgsegm dnn fuzzy img_hash imgcodecs shape videoio xobjdetect datasets highgui superres tracking bioinspired dpm face features2d line_descriptor saliency calib3d ccalib rgbd stereo structured_light videostab xfeatures2d ximgproc aruco java optflow stitching

The key being that “java” and “face” are on the list. Then try to compile :

make -j7

As mentioned if compiling gives you grief try disabling unnecessary modules and make sure your main OpenCV source is as up to date as possible.

You should have a jar in build/bin with face recognition classes under org.opencv.face.

Using it requires the native library to be loaded statically

static{ System.loadLibrary(Core.NATIVE_LIBRARY_NAME); }

I hacked around the library path problem by adding a JVM parameter

-Djava.library.path=/home/edward/Downloads/opencv-3.3.1/build/lib/

Now we should be ready to go! You need to detect faces before learning or recognising them, this kind of thing :

    CascadeClassifier faceDetector = new CascadeClassifier(request.getServletContext().getResource("/WEB-INF/lbpcascade_frontalface.xml").getPath());
    Imgproc.cvtColor(image, image, Imgproc.COLOR_BGR2GRAY);
    MatOfRect faceDetections = new MatOfRect();
    faceDetector.detectMultiScale(image, faceDetections);
    System.out.println(String.format("Detected %s faces", faceDetections.toArray().length));

Next step : train a face recogniser (from a directory of detected/cropped faces in this case)

ArrayList<Mat> sourceImages = new ArrayList<>();
List<String> namesIndexList = new ArrayList<>();
List<Integer> namesIntList = new ArrayList<>();
Files.list(path).forEach(file -> {
    String filename = file.getFileName().toString();
    if (filename.contains("-") && filename.endsWith("jpg")){
        String personName = filename.substring(0, filename.indexOf("-")); // e.g. edd from edd-1.jpg
        if (!namesIndexList.contains(personName)){
            namesIndexList.add(personName);
        }
        Mat image = Imgcodecs.imread(file.toString());
        Imgproc.cvtColor(image, image, Imgproc.COLOR_BGR2GRAY); // Convert image to grayscale or it won't work with the face learner
        sourceImages.add(image);
        namesIntList.add(namesIndexList.indexOf(personName));
    }
});
FaceRecognizer faceRecognizer = LBPHFaceRecognizer.create(); 
MatOfInt matOfInt = new MatOfInt();
matOfInt.fromList(namesIntList);
faceRecognizer.train(sourceImages, matOfInt);

What’s notable here is that when training you pass an array of Mat images and a Mat object representing identifying integers for each face. Hence why I add each new name to an ArrayList so it has an index and then add that index to a list which will have duplicates if there’s more than one photo of a given person. I assume MatOfInt is the right type of object, I found it in the jar rather than any proper documentation. My file naming convention is arbitrary edd-1.jpg, edd2.jpg, ben-1.jpg etc.

If training works predicting is as simple as

Imgproc.cvtColor(image, image, Imgproc.COLOR_BGR2GRAY); // Convert image to grayscale or it won't work
faceRecognizer.predict_label(image)

where predict will hopefully return a useful integer you can use to look up a name. Also worth noting that there are other implementations of FaceRecognizer.

Comments are closed.