The blue gem is inside of the red building. I can’t carry it outside because I need to be holding the red gem in order for the door to unlock. I also can’t drop the red gem while in the vacinity of the door so I am stuck on how to get the blue gem outside so I can bring it over to the blue building.
You need to pick up the blue gem and carry it outside the back garden area of the red House. Drop it on the ground here and then go back inside, grab your red stone and leave. If you go back to the center model, you will see the red stone sitting in the back of the red building and you will be able to pick it up and use it.