Boundlessness is the name of that quality: the fact of having no limit or end. Writers reach for it when they want the feeling of something too big to close — the boundlessness of the ocean, of space, of grief. It is a noun for an idea, not for a countable object; you do not say two boundlessnesses. Quite formal and literary. In speech, people usually just say it seemed to have no end.