Ok, I just measured the circumference of both hats and they are exactly the same (58cm).
It seems to be that the L/XL hat is just physically bigger, the panels seems to be slightly taller and wider. They both have stretchable headbands so they're quite forgiving with regards to different head sizes.
