Yeah, the direction of time doesn't have to be a fact about the laws of physics exactly, it could just be a fact about the broad structure surrounding this particular region of space, time, and possibility. Mountains are pointy and basins are broad and shallow, even if the laws of material transport by erosion are "symmetric" in the absence of gravity.