I'll generally research the issue thoroughly and then discuss options with the client. Communication is the most important part.
For one thing, depending on the problem and the client, the client may very well know others in the industry (it is their industry, after all) who can provide some insight. Sure, I could potentially hire someone myself, and if the client doesn't know someone, then that may be an option, but the problem could literally be resolved by a couple 15 minute phone calls or emails.
Maybe the feature (or method, or software, or component) isn't that important. Maybe there are other approaches that aren't as complicated or better documented or are just merely more expensive. Maybe there's a support team or original developer / engineer who can be paid for assistance. Maybe we can kick the feature down the road and get everything else done while we decide whether the feature is necessary or while we find someone who can help. Maybe there's a training program somewhere in the world that can be visited or invited[1]. Or maybe the issue is core and we need outside help immediately. Figure out the best options available and how much they're likely to cost (temporally and monetarily), discuss them with the client, and then decide together.
The most important point, through and through: Research Options and Communicate. You're being paid for your expertise, which is to say for what you know and for what you're prepared to learn due to your experience (and schooling), and part of that expertise is the ability to say you need help.
It's far worse and terribly unprofessional to waste valuable time and money getting nowhere than risk losing the project.
1: Story time: A friend's company recently gave a full-time employee two weeks to try to figure out a CNC router she bought from China. The employee is on a salary and it's their slow-season, so she didn't lose too much in letting him try. Unfortunately he couldn't figure it out, and now she's hiring an expert from out of state to come spend four days training her team. That training cost half as much as the CNC, but once her team understands it, they're going to make many times that per month. If I spent that much time trying to figure out that CNC at my daily rate, I would have far surpassed the cost of the router and potential profits within the first couple days - which would have been incredibly wasteful for all concerned parties.