Lines eleven to fifteen introduce us to the word death. The writer says that what they talked of was of no great importance, which shows his realization of the shortness and immateriality of life. In life death cannot be avoided. In his case, it would seem, there would be more than one sort of death. One would die physically, the other emotionally.
The next five lines show how small a difference every life makes in the greater scheme of things. The hill, portraying the earth, is climbed by thousands. Yet, to the individual (him) the 'minute' of a single life is of immeasurable importance. 'Foot-swift, foot-sore' symbolizes the haste in living, and also the pain. People strive to get to the top, ruin their lives to get there, possibly, but upon arriving see there is nothing.
The following five lines are lighter than any have been before; yet they are filled with irony. The primaeval rocks (the world, history) have been shaped and given "colour" by many, many passers-by. The writer, however, illustrates individual selfishness by saying that it was him and his lover or wife that gave the shape and colour to those rocks. He is looking back in irony, seeing now, perhaps, how his views have been changed.
Lines twenty-six to thirty are a sad depiction of reality. The writer states that Time is severe and harsh, that it has dealt with him mechanically, without any thought given to his emotions and feelings. The substance now, as he calls himself, is only one phantom figure. By calling himself a substance he could be comparing himself to a substance used in scientific experimentation. He has, he feels, been used. Now he is merely a phantom. All life has been taken from him, and all that remains is his physical form. He says that he remains on the slope, meaning that he has not moved on, or that he cannot move on. The loss of another has left him immobile, lost, or simply without direction.
The last five lines shows a picture of the writer driving away from himself. Rain is mentioned again, like tears falling. The person on the hillside is a vision of the man he used to be. That man is "shrinking and shrinking" and the writer is moving on, changing. The drizzle of the first five lines has turned into full-on rain, and this makes the person even more dim in the writers memory.
The ground he thought was solid is sinking away ("for my sand is sinking"), and he comes to the realization that he will never be able to go there again. He calls the "there" old loves domain, which means that he is moving on, perhaps to new love.
The final line is abrupt, the shortest in the entire poem. This shows how quickly everything happened to him; without warning, all was gone.