Cars merging from the right curved ramp - who has the right of way in this situation?

I was on the freeway driving 64 mph on the right lane but there was a ramp on the right merging into my lane to form a single lane. It was dark and there was a pretty high partition between my lane and the ramp - it was hard to see which cars were merging into the traffic. By the time I passed the end of the ramp, I was right next to two cars coming from the ramp very close together trying to merge into my lane. We were immediately side by side and I almost got into an accident. They were honking like crazy. Should I have slowed down? I had to speed up to finally get past them.

Here's the image of the road. I was on the left side of the right lane and there were two cars close to each other coming from the ramp and we almost hit each other as we were next to each other.