The worst thing that could happen is you fail and lose a lot of money.
That's not so bad. Anyone willing to take a risk in life is willing to both fail and lose money (sometimes theirs, sometimes someone elses). I've failed MANY times over... a few times in a BIG way. I've lost more (of my own) money than it would take to stay on the mini tours for 10 years... and I'm a better person for it.
Don't get me wrong, failure SUCKS! It's depressing, makes you question yourself, makes you question your abilities, breeds doubt, keeps you up at night, etc. So you have to be prepared for that going in.
Depending on someone's life situation (kids, wife, etc) I think the decision making process is pretty simple: Provide for your family as best you can and hold onto your dreams as much as you can with the spare time/ money you have. If you have no family to provide for, then provide for yourself and CHASE your dreams as much as you can with the spare time/ money you have.