My gut feeling is that since the main flow is with rollback it does not matter what the sublow does because the whole flow (including the subflow) will be rolled back. But I would have to make a testcase to verify it.
There is quite extensive documentation in the Mendix HowTo - Set Up Error Handling section.
According to that, if error handling is set to rollback, everything up to that point will be rolled back.
EDIT: creating logging data can be problematic indeed. Any exception happening afterwards may rollback your error data. I'd suggest to use standard Logging instead.
Had the OrderLine object in the subflow been some independent object, it would have been committed.
But the OrderLine is associated to the Order that is being rolled back, hence the OrderLine is rolled back too.
It is a scenario that indicates why you should be very careful with these mixed error handling scenarios.