If he is growling I'd not remove him and then cuddle or hug him! This is potentially giving him a reward for his unwanted behaviour. Remember he connects your behaviour and response to his recent behaviour. Instead I'd be luring him off the couch and rewarding with treats when HE HIMSELF makes the right decison -- to get down without growling at your request. If you just lift him off then reward him he may well accept that while you have moved him somewhere, you are then immediately signalling he is still more important than you.
You neither want to inadvertantly encourage the unwanted behaviour nor to put yourself at risk of this behaviour escalating. There's a link at the end of this post to a lab rescue article that directly addresses the couch situation and that will probably give just the guidance you want.
With more than one dog in the house I'd personally be wary of letting others have access to couches etc and barring one dog -- either this goes for all or for none. I know that some trainers believe there's a correlation between allowing dogs to be 'elevated' and use furniture but personally I side with those who think this is nonsense.
The better behaviour that sometimes comes with barring dogs from particular places (bedrooms, sofas) is due not to refusing to allow a small dog to be at eye level, but that people start to use a 'no free lunch' approach of requiring general polite behaviour from their dog -- which includes requiring a dog to have permission to get on furniture if you allow this. That all helps reinforce wanted behaviour generally. ALL the dogs should be required to show polite behaviour with getting up on furniture.
Are you doing other basic obedience with him to reinforce your leader role in the house? For example to require sits and downs before good things happen (eg being invited on to a lap? getting his meal? A treat?). Do you ask him to sit politely while you go through doors? Working on some basic obedience consistently in everyday activities tends to really help with this kind of thing. The reason he is not growling at your husband and is growling at you is likely down to how your husband and you interact with the dog. I am guessing your husband is probably less inclined to give him lots of attention in ways that are telling him he runs the place. This is a very tricky area -- our inclination with rescues is so often, and so naturally, to make up for all they have gone through before. You really need to remove that notion completely and instead, focus on the dog you have now and the dog you want him to become. If you cater to him, give him special attention, begin to act decisively (eg remove him from the couch for growling) but then show appeasement TO HIM for your behaviour (cuddling him right after, which to the dog, says, I recognise you run things, let me show you that you are really the important one) then he is learning that you are further down the line than your husband (the boss ) and him (number two).
It is really important to give these dogs structure and reward thru the very act of expecting and asking for good behaviour, all encouraged through rewards and praise. But likewise, not to overly indulge or fuss over the dog -- which to the dog, means you are the one acknowledging its new superior position. When rescues come into the house they are generally very cautious and wary as they settle in and look for cues as to where they fit in. If your cues are to give lots of attention to the dog the chance is that the dog assumes this is the way it will always be -- you acknowledging the dog is more important than the other dogs and cats and people in the house thru such behaviour. Hence a lot of trainers suggest giving calm, but slightly indifferent attention to a rescue for the initial weeks, allowing it to find its position and feel comfortable in the home and gradually fit in. And to always start as you mean to contuinue -- don't give the rescue special treatment and privileges (eg a free lunch) then expect compliant good behaviour later on. You need to set kind but firm structure from the start -- and it's never too late to start.
I have a lot of links on all aspects of behaviour and rescue dogs here:
http://www.cavaliertalk.com/phpBB2/viewtopic.php?t=1705
Have a look at:
http://www.deesdogs.com/documents/thereisnofreelunch.pdf
This advice is excellent and I have seen it over and over and over again in rescues:
All rescue dogs go through a "honeymoon period." After the first day or so, the dog may be very quiet and extraordinarily controlled and "good." The "real" dog appears two to four weeks later - after he's mostly figured out the house rules, the schedule of the days, and the characters of his new family. At this time, he'll start testing out his position in the pack, and may "regress" to puppyhood behaviors and "bad" behavior. Be patient with him, firm in your expectations, praise him for appropriate behavior - especially when he is lying quietly and behaving himself. Don't praise for nothing constantly - the dog will learn to tune out your praise over time!
which comes from:
http://www.cbrrescue.org/articles/bringhome.htm
The lab rescue link is excellent on a whole range of issues, but you will especially want to read:
http://www.labadoption.org/LinkPages/DogBehave/Articles/QueenOfTheCouch.pdf