Should I break up with my fiance?

We are not formally engaged but have been together for four years, lived and studied together, met each others parents and plan to get married one day. We have a very happy relationship. We never fight, we have the same interests and similar points of view, we are both sweet and caring and we are each others best friend.. The only problem is that our relationship lacks sexual passion. I mean, we do have sex, but something is missing there. He doesn't seem too crazy about it, like I thought people in love should be. I don't know whether its his personality, or is he not that attracted to me physically. It started with a health problem I had in the first year of our relationship, that made it hard for me to have sex. I'm ok now, but he always blames my illness whenever I complain about our lack of passion. I don't think its just this, it would have changed by now. Our love is more caring, sharing and being there for each other, rather than sexual attraction, especially for him. I was always crazily attracted to him, but most of the time I was too embarassed to express it when he didn't seem to feel the same way. Whenever I start passionate kissing/sex, he loves it, but I have to start it. Recently I've met a guy at work who I'm very attracted to, and he seems to be feeling the same. I haven't cheated but i'm falling for him. It's not just because I'm tempted to date the new guy. I think that having a crush on someone else is an indication that something is missing from my relationship. I wonder whether I should break up an almost perfect relationship... I know I will never meet a guy like my partner again. But can I live forever without a fullfilling sexual life? What do you think?