You should go. I know from experience that if you have a special bond it is important that he hear your voice and sense your presence. Perhaps to not offend your grandmother you could arrange to meet her somewhere (restaurant or coffee shop) as when she sees you in person she might feel relieved rather than upset. My Mom had my father at home with hospice and kept insisting that she could do it all and he wouldnt want to be a burden, etc. but I was very persistent and the day he passed away was there all afternoon-- it turned out I was badly needed particularly right after he passed. Will your grandmother have anyone to turn to in the hours after your grandfather does pass away? She needs you and I'm sure your grandfather would want you to help her as well.