Depending on your play style, this may or may not be something you need to worry about. Destruction magic covers all of the magic types that can deal damage. If you are like me and prefer to go down the road of swords and shields, you might find that your destruction magic abilities are a little low. A less grindy way to level this up is to go to some dungeons or locations that contain enemies of a much lower skill level. Perhaps some of the early level areas that you skipped. You will have enough health and armor to absorb a lot of damage so you don’t die and you can then focus on killing the enemies with your destruction spells in order to level it up.
If you don’t feel like trecking around then you can also grind this. If you get lucky you might be able to find a hut or something with a door that can only fit a person, you can lure some highly powerful and large enemy, such as a giant to this location and he will get stuck outside. You can then attack away from inside, dealing damage without getting hurt. Another method people like is to summon a frost atronach or some other creature that you can summon using magic. Summon it and then kill it using destruction magic. You can then level up both conjuration and destruction at the same time. It will cost you quite a bit in magika though.