A security group can be compared to a user role. For example, when a company is split into several departments and members of a department may only access the orders for their department, you can use a security group.
Example: Metaobject "Department" has an association to "User" (which user is in which department) and an association to "Order" (which order belongs to which department). After creating a security group for the Department object, you can configure instance access on the Order object to check for the Department security group.
Instead of using security groups, it is also possible to use the XPath Constraints option in the instance access dialog to achieve the same effect.
Security groups can be made for objects that are related to the User object in the System module.
Once set, this can be used to set instance access for objects that have a relation with the object used in the security group. All you have to do now is set the active flag.
Example setup: An Account object has a relation with System.User object. A security group is added using this Account object and its relation. An Email object is related to this Account object. When setting instance access for this Email object, you can select 'active' in the part 'Associations to security groups' and from now on, users that have a specific account with a relation to a specific email can see this email.
As benny says, you can use xpath constraints to get the same result. But clicking is off course much easier and there is no risc that you type some xpath queries wrong