I think you're right to be cautious. If it does work out, you would be exposing your children to him also so I don't think there's any reason you shouldn't be completely comfortable with this guy.
He does sound a little 'needy'. That might not be so bad if he's getting what he wants but his reaction to your answer shows he might want more than this later on. Does he have kids? Do you think he'll understand that your kids will always be a higher priority than him? That might be tough.
Maybe you should sit him down and explain that you need things to go a little slower than that, and that you will probably be more cautious than other women he's been with? If he seems genuinely agitated, it might not be a good match, you know? Then you can
him with a good conscience.