[Modified from something I tried to post on 538, but there were technical difficulties.]
The belief that new jobs will always replace old jobs is misguided in many ways.
First, unemployment is much higher than the official figures let on, and the true surplus of non-working, potentially productive people has been rising for decades. People spend more of their lives in college. People spend more of their lives in retirement. We have more people on government disability, more people in prison, more part time workers, and more people leaving the job market entirely.
Those jobs that disappeared didn't all come back.
Second, when you automate all the manual labor jobs out of existence, and replace them with more mentally taxing work, there are going to be millions of people who were perfectly capable of holding the old jobs, who can no longer provide the engines of capitalism with any service it's willing to pay for.
What is the free market answer to their plight?
1) They should pull themselves up by their own bootstraps. Start your own business! Be dynamic and innovative! Reach for your dreams! Or failing that...
2) Die. Survival of the fittest, you know.
If you don't have a better answer than that, then you cede that capitalism + technological progress can bring great misery and suffering to those made obsolete by technology.
We've reached a point in our technological climb that is somewhat analogous to the situation we have with oil reserves. New reserves of jobs are being found, but the old jobs are being depleted at a much faster rate.
Take the precarious position of my bro. His job at the postal service is to sit in front of a computer all day, looking at pictures of individual pieces of mail, and routing them appropriately. There are thousands of people similarly employed at routing stations across the country.
Or, at least, there were. As the Post Office's handwriting recognition software has improve, the volume of mail that needs manual routing has fallen dramatically, and center after routing center has been closed, their employees released into the wild to make their ways as best they can. Now only two or three centers are left.
Nor is it just repetitive tasks that are being obsoleted. Software development and deployment has gotten easier in a variety of ways, driving down the costs of bringing new ideas to market. A web app that might have taken a team of a hundred people a year to deploy back in 2000 could be done by a team of three people in a couple of months today. Even Facebook (a site with hundreds of millions of active users) only employs about 800 people.*
I'm not sure where the next big "growth industries" are, the ones that are going to absorb all the medical transcriptionists, all the long haul truckers, all the taxi drivers, all the delivery people, all the tech support and customer service representatives, all the janitors, all the airline reservation people, who are going to be made obsolete over the next few decades.
I see the march of technological progress as a good thing. Nobody is asking that we halt the march of technology to save jobs. But unless we find some way to "spread the wealth," to ensure that everyone can have some claim on the products of a highly automated economy, then we really will hit the crisis point that Marx predicted.
A compromise between socialism and capitalism could be forged in several ways. Guaranteed income floor, make-work jobs, wage subsidies, etc. Right wingers will fight all these measures, right up until the moment that their own jobs go on the chopping block. Then they'll see the benefits of "institutionalized theft."
* It would be 400, but they have trouble keeping their employees from goofing off on Facebook.