There a few different ways you can accomplish this.
Here's one example:
You can maintain a count and a flag on the class entity that indicates if the class is full or not. You can do this by using a Before Commit event on class the class entity that updates the attribute for count, and determines if the flag for full should be set to true. This way when you display classes, you can use an xpath constraint to only display open classes, and easily know in a microflow if a student can enroll in a certain class.
Here is some documentation on event handlers. You would want to retrieve all students by association from class and use a list operation to count the list. As well as logic to update the student count and when to set the enrollment full flag to true.
Also here is the link to the learning paths. If you go through all the modules, you would have more than enough knowledge to accomplish what you are trying to do.
Hope this helps!