Modern Web Weekly #5

It seems like face detection is finally coming to a browser near you! 🎉

In an experimental phase to be clear, but now that Chrome Canary, Edge Canary, and Safari Tech Preview 172 all support face detection it seems this is slowly becoming a reality.

So how does it work?

Detecting faces with the Shape Detection API

Face detection is part of the Shape Detection API which also includes barcode detection and text detection. The BarcodeDetector object has already been supported by Chrome on MacOS, ChromeOS, and Android and Edge on MacOS. The FaceDetector object is now supported in the Canary versions of Chrome and Edge and Safari Tech Preview 172 which was released last week.

In all these browsers it's available behind a feature flag, which means that in Chrome and Edge, you need to visit chrome://flags and edge://flags respectively, look up "Experimental Web Platform Features" and set this to "enabled". In Safari Tech Preview, go to the "Develop" menu, then to "Feature flags" and then check the box for "Shape Detection API".

You can then detect faces in images or video streams like your camera. Note that this API only detects faces, not who a face belongs to.

Working with FaceDetector is surprisingly simple. It has a detect method that takes an HTMLImageElement, a Blob or an ImageData object. This means you can use an <img> element, an uploaded file (Blob), or the video stream coming from your webcam.

When using your webcam, you can continuously draw a frame of the video stream from your webcam to a <canvas> element and then get that frame as an ImageData object using its getImageData() method. You then give that ImageData object to the detector's detect method:

const detector = new FaceDetector();

// get the ImageData object somehow
const imgData = getImageDataFromCanvas();

const faces = await detector.detect(imgData);

The FaceDetector returns an array of result objects (faces) that have boundingBox and landmarks properties. The boundingBox is a DOMRectReadOnly object that has the properties x, y, width, height, top, bottom, left, and right and represents the coordinates and dimensions of the box around the area in which a face was detected. You can use this information to draw a box on the <canvas> element around the detected face.

The landmarks property holds information about the facial features of the detected face like eyes, nose, and mouth. It consists of an array of objects that have a type property that represents the facial feature ("eye", "nose", or "mouth") and a locations property that holds an array of coordinates (objects with an x and `y property) that indicate the position of the facial feature. You can draw a line from the first coordinate and then to all subsequent coordinates to indicate the detected facials feature on the <canvas> element.

Here's an example (shortened for clarity):

{
    "boundingBox": {
        "x": 278.80908203125,
        "y": 186.04685974121094,
        "width": 113.48743438720703,
        "height": 113.48743438720703,
        "top": 186.04685974121094,
        "right": 392.29651641845703,
        "bottom": 299.53429412841797,
        "left": 278.80908203125
    },
    "landmarks": [
        {
            "locations": [
                {
                    "x": 307.5032043457031,
                    "y": 215.3684539794922
                },
                {
                    "x": 312.7261657714844,
                    "y": 212.50616455078125
                },
                . . .
            ],
            "type": "eye"
        },
        {
            "locations": [
                {
                    "x": 365.8408203125,
                    "y": 217.3159942626953
                },
                {
                    "x": 361.17852783203125,
                    "y": 213.92254638671875
                },
                . . .
            ],
            "type": "eye"
        },
        {
            "locations": [
                {
                    "x": 325.0015563964844,
                    "y": 263.11260986328125
                },
                {
                    "x": 329.6048278808594,
                    "y": 261.1650695800781
                },
                . . .
            ],
            "type": "mouth"
        },
        {
            "locations": [
                {
                    "x": 339.57855224609375,
                    "y": 221.32908630371094
                },
                {
                    "x": 327.9523620605469,
                    "y": 235.07989501953125
                },
                . . .
            ],
            "type": "nose"
        }
    ]
}

I added a face detection demo to What PWA Can Do Today, a showcase of what is currently possible with Progressive Web Apps. This showcase is itself a PWA so you can install it to your phone or desktop a check for yourself what your device can do.

Check the demo at https://whatpwacando.today/face-detection

I also created a codepen that you can play with yourself. It shows the face detector implemented as a web component. It uses two <canvas> elements: a hidden one to draw the frames from the video on and another one that's laid over the <video> element that the boxes around the detected faces are drawn on.

The source code is heavily commented to explain all the parts.

Here's the codepen (open the browser version of this email if it doesn't show in your email client): https://codepen.io/dannymoerkerke/pen/KKrgGgy

Anchor Positioning comes to Firefox

In Modern Web Weekly #2, I wrote about CSS Anchor Positioning, a new API that enables web developers to make sure a popover (menu, tooltip, dropdown, etc) is always displayed inside the viewport using only CSS.

Currently, this API is only implemented in Chrome and Edge Canary behind a feature flag but now the Firefox team showed interest as well and has started implementing this feature already!

Here's the Github issue to track progress: https://github.com/mozilla/standards-positions/issues/794

A better install UI for PWAs

This article was featured in the first edition of Modern Web Weekly I mistakenly only sent to one subscriber: me. To make sure you won't miss it, here it is again:

If you install a PWA, you can now add a description and screenshots of the app to the manifest.json file which will be shown to users when they install your app, similar to what you get when you install a native app from an app store.

You do this by adding a description field and an array screenshots to manifest.json like this:

"description": "My awesome PWA", 
"screenshots": [
    {
     "src": "source/image1.gif",
      "sizes": "640x320",
      "type": "image/gif",
      "form_factor": "wide",
      "label": "Wonder Widgets"
    },
    {
     "src": "source/image2.gif",
      "sizes": "1280x320",
      "type": "image/gif",
      "form_factor": "narrow",
      "label": "Wonder Widgets"
    }
]

What wasn't immediately clear to me is that images specified in screenshots with form_factor: narrow are only shown on mobile phones and images with form_factor: wide are only shown on desktop.

I wasted some hours trying to find out why I didn't get the improved UI on desktop until I realized I had only specified images with form_factor: narrow.

There are a couple of other criteria that screenshots must follow and Adriana Jana has all the details.