- Create the Selection outside of the Select and Mask Workspace
- Use the Overlay View Mode on 40% opacity. Change the colour of the Overlay if necessary.
- Use the Refine Edge Brush Tool with Show Edge ticked.
- Tick and un-tick Show Edge until you think the image looks ok.
- Try not to use Decontaminate colours.
- Output as you like.
I have a video on the Select and Mask Workspace
What is a Mask
There are three types of Layer Masks, Clipping Masks, Alpha Masks (Channels). Bottom line a Mask: hides or reveals parts of an image. Black conceals white reveals. And, grey partially reveals.
What is a Selection
To Photoshop a Selection is a Mask they just appear different to the user, but under the hood, they are the same thing. As we go forward I will use both words interchangeably.
With a selection, you see a marching ants border. If you saw pure black, for instance, you would not be able to see what you’re selecting.
Select and Mask Workspace Overview
The Select and Mask Workspace is a suite of Tools that can create selections and masks.
You can open the workspace with or without an existing selection.
Although you can create a selection from scratch in the workspace there are fewer tools for mask creation inside the workspace, compared to outside of the workspace.
What sets the workspace apart from any other selection tool in Photoshop is the ability to refine the edge of a selection.
- Quick Selection (W): Tool is a quick way of creating selections based on colour and texture similarity. The tool has no Auto Enhance tick box, like the normal Quick Selection Tool. Auto-Enhance is on by default here. Use it like a normal brush by clicking and dragging to select areas.
- Refine Edge Brush Tool (R): With a selection or mask already active. You can use this brush to include areas previously not selected at the edge of the current selection. Every time you click and drag (paint), Photoshop will assess the area painted and attempt to refine the mask edge to exclude the background on let’s say fine hair. This is the most important tool and is often used with “Show Edge” ticked. You can add or subtract from the mask by using the Options Bar, or by keeping Alt/Option pressed.
- Brush Tool (B): You can add or subtract from the mask/selection. With the Brush Tool, you have full control over what areas are selected or deselected. You can add or subtract from the mask by using the Options Bar, or by keeping Alt/Option pressed.
- Lasso Tools (L): You can add, subtract, or intersect with the mask using either the Lasso Tool or the Polygonal Lasso Tool. By
default,you are adding to the mask. You subtract (Alt/Option), or Intersect (Shift + Alt/Option), with the current mask.
- Hand Tool (“H” or Spacebar): Used to move around when you are zoomed in.
- Zoom Tool: (Z): You can use Alt/Option to pressed to zoom out. Ctrl/Command 0 to fit on the screen. Ctrl/Command 1, for actual size.
- Show Edge (J): THIS IS IMPORTANT: This will show your edge. But, first, you have to either: Set a radius with Edge Detection. Or, paint with the Refine Edge Brush Tool. If you have done neither the preceding you will not see an edge.
- Show Original (P): Will show the original selection (before you entered this workspace). This is best seen in the “Marching Ants” View.
- High-Quality Preview: Renders an accurate preview of the changes. This option may affect performance. When this option is selected, while working on the image, hold down the left mouse button (mouse down) to view a higher-resolution preview. When this option is deselected, a lower-resolution preview is displayed even on mouse down. Best not used, it affects performance significantly.
- Opacity: Only available on certain views. Dictates the transparency of the overlay.
View Mode settings
If already have the layer below to which I intend to be the background of my masked image – I use On Layers
If I don’t have the layer below to which I intend to place my masked image on – I find Overlay on a reduced opacity – like 30% to be most useful.
View Mode. From the View pop-up menu, choose one of the following view modes for your selection:
Onion Skin (O): Visualizes the selection as an animation-style onion skin scheme. Better seen than explained.
Marching Ants (M): Visualises the selection borders as marching ants, as per a normal selection.
Overlay (V): Visualizes the selection as a transparent colour overlay. Unselected areas are displayed in that colour. The default colour is red.
On Black (A): Places the selection over a black background
On White (T): Places the selection over a white background
Black & White (K): Visualises the selection as a black and white mask
On Layers (Y): Surrounds the selection with areas of transparency so you can see any layers below if any. Probably the best view if you have the layer below on which you intend to composite.
Press F to cycle through the modes and X to temporarily disable all modes.
Edge Detection Settings
It is not always necessary to use the Edge Detection settings as you are basically trusting photoshop to detect the width of your edge. Better results are often obtained using the Refine Edge Brush Tool with Show Edge ticked on.
The is the semi-automatic way of telling photoshop how wide you want the border of your mask/selection to be.
As you increase the radius it causes photoshop to evaluate what’s inside of that border.
Ticking Smart Radius works like this. Say you had a border around some flyaway hair, and also some clothing. Smart radius would not increase the border around the clothing as it has a smooth outline, but it would increase the border (according to your radius setting) around the hair as it has an irregular edge.
Global Refinement Settings
- Smooth: Reduces the irregularity of the edge.
- Feather: Softens the edge of the mask.
- Contrast: Increases the contrast of the edge reducing the softening. This is the opposite of Feather.
- Shift Edge: Moves soft-edged borders inward with negative values or outward with positive ones. Shifting these borders inward can help remove unwanted background colours from selection edges.
- Decontaminate Colors: Replaces colour fringes with the colour of fully selected pixels nearby. The strength of colour replacement is proportionate to the softness of selection edges. Adjust the slider to change the decontamination amount. 100% (maximum strength) is the default value. This will limit your Output options, you cannot use Selection or Layer Mask. Decontaminate will also make changes to the image layer itself, you notice blockiness around the edges, though the layer works overall it’s strange to see this colour blockiness.
Selection and Layer Mask are not available when Decontaminate Colors is used.