I’d set the date to a fixed value for the entity holding the times.
As the date is counting from 1-1-1970 maybe use that for the times then the data in the db will only hold the seconds or ms from 0.Then sort on the times.
As your time is always respective with associated date entity value, you can calculate the time value as suggested by Erwin in time attribute. To add to it, I would also consider timezone settings during implementation and showing these values on pages as time is crucial part here.
Also make sure of testing edge cases to check time values when timeline spans to next day.