It's absolutely true that our recorded experience of the Earth's weather patterns is extremely limited. It's also true that through studies of geology, tree ring signatures, ice cores, and like, we see strong evidence that the Earth has had many, many cycles of hot times and ice ages.
Most scientists I've read on the subject agree that there are typical cycles and that we are likely in a cycle that is moving from cold to warmer since the last ice age. They also think that as the Earth gets warmer it typically then gets colder soon after because of evaporation, clouds, changes to winds & ocean currents, etc. Many scientists also believe that this trend does occur naturally and independenly of man. That said, many believe that we are accelerating it, and that's the big concern. They're worried that the acceleration could trigger large-scale changes more quickly than they would normally happen. And, faster changes mean that life might not have enough time to adapt.
For example, there's pretty strong evidence that a ~5 degree increase in the temperature of our polar waters could seriously impact the yearly cycles of algae/plankton blooms that form the basis of the ocean food chain. If this occurs over, say, 12,000 years, the algae/plankton would likely adapt and still bloom, or even move and bloom elsewhere. If it happens over, say, 100 years, the algae/plankton might not be able to adapt, and the snowball effect of such a loss could cascade through the entire ocean and land-based ecologies disrupting semi-stable systems in a catastrophic way.
So, the issue is not that we don't know about and expect *natural* global warming, it's that we don't know what happens if said global warming happens more quickly than it would naturally. And, as "they" say, what you don't know can kill ya.