In the lord of the rings the return of the king does frodo actually die at the end?

Yes- he does die. When he's going off to the gray havens with the elves in "the last ship to leave middle earth" he is going to heaven. If it's leaving middle earth, it's not a place Sam, Merry or Pippin can ever go visit him, and that's why they're all so upset. He was just too broken from the whole experience, like he says at the end “How do you pick up the threads of an old life? How do you go on, when in your heart you begin to understand there is no going back. There are some things that time can not mend. Some hurts that go too deep... that have taken hold.”