Pokemon Gyms are scattered all over the world. Typically they are located in significant locations that are more important than something that could be set as a Pokestop. You must be at least level 5 before you can take part in any gym battles. Once you reach level 5 you can join a gym or battle the trainers by walking to the location and tapping on the gym on the map. Gyms & battles are a key part of Pokemon Go outside of actually collecting Pokemon. This wiki section will give you a good overview of how the gyms work so you can start working your way up to become the top gym leader in your area!
Depending on which of the 3 game factions are in control of the gym, the color of the gym will change. When a gym is in control by your faction you can join if your Pokemon is powerful enough to win a place on the team. If the gym is owned by another faction you will get to pick 6 of your most powerful Pokemon to use in the battle against all of the Pokemon trainers currently defending the gym.
Once you become a trainer in a gym you will earn 10 Pokecoins every 21 hours so long as you remain in control of the gym. These coins can be redeemed from the store menu, by clicking the shield icon in the top right of the screen.