This mask is defined through a polygon and a feather parameter.
To edit the points of the polygon, you need to use the Edit Mode Allow to edit quickly some layers parameters Read More of the Viewport Where you see what is happening in Smode. You can do without it but what would be the point? Read More :

The Edit Mode Allow to edit quickly some layers parameters Read More is relevant to the currently selected element in the Element Tree The hierarchical view of documents in Smode Read More , so take care that the polygonal mask is currently selected. Now, press [CTRL] and click in the Viewport Where you see what is happening in Smode. You can do without it but what would be the point? Read More to create points:

When there are more than two points but the polygonal mask is not closed, it will display a 50% overlay on the whole image. To close the mask, [CTRL] + [CLICK] on the first point:

The points of the polygon can be moved at anytime by dragging them in the Viewport Where you see what is happening in Smode. You can do without it but what would be the point? Read More . The only necessary thing is to be in Edit Mode Allow to edit quickly some layers parameters Read More with the polygonal mask currently selected.
To animate a polygonal mask, just drag and drop the "polygonal mask points" parameter inside of the Timeline Timelines specificity is to play, cut and edit video layers easily, and to animate layers and their parameters Read More . Now each key you'll add on this parameters will concern every points of the mask :


You could also animate all the mask by using Exposed Parameter Allow to control multiple Link Target with a single value Read More on the "Points" parameters.