I've really hesitated to weigh in on this. I feel enormously ill-equipped to provide the kind of answers these questions and thoughts ask for.
I'm not going to sit here at my computer and casually type "I know what you're feeling". I have no idea. For whatever reason, God has protected me from degrees of suffering that others deal with nearly every day. My concern is that the lack of comparable pain in my life will make any words I have to say feel empty.
I think many times, when we ask the kinds of questions you are, we don't need answers as much as we need someone to listen. Of course other times, when the emotions are less fresh, we really do want to engage these kinds of issues as logically as possible.
If what you could most use is someone to listen, Rickster, I'd be happy to do that and be there for you however I can. Shoot me an e-mail and let's connect some more.
In the meantime, I'll put some thoughts out on the table that have resulted from my own searching on this kind of subject.
First, I think there is some popular teaching out there that sets up some very harmful expectations. Some might label it the "health and wealth" gospel, where if your faith is strong enough you can spread your jazz hands with Annie and sing "The Sun Will Come Out Tomorrow".
I don't subscribe to this view, because I don't think it reflects scripture accurately. I'm always open to new information, but I haven't found scriptural support for this view yet.
Jeremiah 29 was written to the Jews in exile before the birth of Jesus. It was not written as a blanket promise for happiness in this life for all believers. Words that can be more easily accepted to be for all believers would be those of Jesus.
John 16:33- I have said these things to you, that in me you may have peace. In the world you will have tribulation. But take heart; I have overcome the world.
I hate verses like this one, to be honest. At least I hate that middle section. I want to believe that the things I'm working toward will succeed in the way I have in mind. I'm repeatedly frustrated when God seems to be sending me in a clearly defined direction and yet I slam into wall after wall trying to get there.
A number of years ago when I accepted the position of worship pastor at my church, I had a clear vision to change the focus of our sunday morning worship time from one that was focused on music and performance, to one that was about connecting people to the words we were singing and the purpose for coming each Sunday. The leadership of the church had a similar vision, although we didn't know what it would look like.
Some of the "old-school" vocalists were frustrated by the seeming lack of direction we had as we moved forward. But I shared with them a little nugget from scripture that, while unable to clarify our direction, helped adjust some of our expectations.
The passage is from Acts, Chapter 16, specifically verses 6-8. Many moons ago I covered this on the blog and podcast and wrote:
Something interesting happens in verses 6-8. Paul and Silas chose the area of Phrygia and Galatia, not because God clearly said, "go there", but because he told them where NOT to go. The rest was educated and prayerful guessing. They saw Mysia coming up and knew it was technically part of Asia Minor(which they were supposed to avoid), so veered away to Bithynia in the north. But Jesus didn’t want them there, either.
At times, these great men, in line with God’s will, were confused wanderers, having NO idea what was coming next, but trusting that they were in the will of God because they kept their eyes on the mission God had given them: Create disciples, who would create disciples, who would create disciples… Very often we don't get the specific direction from God that we would like. "Should I quit this job? Sell this house? Start this ministry?" We may try things out that we believe are in the will of God, only to see them fall apart. We can take comfort in knowing that this happened to Paul, too. And while it may seem less than productive to us, these aren't "detours" from God's plan. Paul used what knowledge he had available to make wise decisions and made serving God and others his number one priority. He hit some U-turns, and so will we, but this is a simple formula that anyone can honor God with.
I always get internally concerned when someone says that "God told me such and such". I want to find a tactful way of asking "How do you know the difference between what God says and what thoughts your own mind creates?"
The truth is, if we are looking for promises either in scripture or from the "voice" of the Holy Spirit, we shouldn't expect those promises to be for good things in life, like a spouse, children, a good job, financial security, etc.
Coming back to Jesus' words:
John 16:33- I have said these things to you, that in me you may have peace. In the world you will have tribulation. But take heart; I have overcome the world.
Jesus only promises here that we will have pain in this life. But his desire, and his expectation given what truth he has revealed and his presence through the Holy Spirit, is that we "may have peace". Peace in this verse doesn't mean "no pain", but rather "contentment" even in the midst of pain.
Part of that contentment, I believe, is meant to find its source in the back end of the verse. He says "take heart; I have overcome the world."
Part of me wants to say, "well good for you, Jesus. Lah-dee-frickin-dah. What about ME?"
The answer, I think, is in 1 Corinthians 15:20-
But in fact Christ has been raised from the dead, the firstfruits of those who have fallen asleep.
Because Jesus overcame the world, overcame the worst it had to offer, including death, those who place their trust in him will be raised like he was, devoid of pain, hurt, frustration, and unfulfilled longing. And in this state they will thrive forever, making the 80-some years spent on earth an eternally shrinking dot on the line of eternity.
When I was leading worship I asked my senior pastor what kinds of songs we should sing more of. His answer surprised me. He said, "we don't sing about heaven enough." And he's right.
I live my life constantly with the narrow focus of these few years i have on earth. I find that when I feel depressed, unfulfilled or frustrated with my situation in life, I can source it back to having a narrow view that forgets to factor in eternity.
I struggle with having that view every day. Just a few hours ago I read a disappointing review of something I created (it was actually a pretty good review, but I tend to focus on the negative). After an hour of sulking with a pit in my stomach, I just surrendered to God my need to be a good artist, or "the best at what I do", reminding myself that in eternity I will be able to express myself creatively without any of the insecurity and discouragement that comes with it right now.
To this point, most of what I've written has been related to "dealing with disappointment". But the other implied question you asked was "How do I know what God wants me to do?"
I'm assuming you aren't referring to the general answer to that question, provided by scripture, but the more specific answers about what God wants or what he has planned for you.
At this point, because I don't have the confidence to take you through the subject from scripture, I'll refer you to a three-part series that my pastor just finished this Sunday on listening to the Holy Spirit. (I can't help but wonder about God's timing in this, given your post and my own wrestling with this issue lately.)
If you're interested, you can download each part here:
Part 1- http://s3.amazonaws.com/churchplantmedia-cms/redmountaincommunity/tw_04.mp3
Part 2- http://s3.amazonaws.com/churchplantmedia-cms/redmountaincommunity/tw_05.mp3
Part 3- http://s3.amazonaws.com/churchplantmedia-cms/redmountaincommunity/tw_06.mp3
Feed located at- http://rmcchurch.org/feeds/sermons
I hope something in all of that is helpful to you.