Some people say we have too many people on the planet, and some say population decline is just as bad? Which one is it?

I understand that population decline creates stress due to the change of balance between forces, but it is definitely a bad thing?