The Blues held a 31-point lead over Port Adelaide in the second quarter on Friday night only to slip to a 14-point defeat at Marvel Stadium.
Carlton have lost three of their past four matches, while their midfield are lacking the spark they had early in the season.
Patrick Cripps, Sam Walsh and George Hewett battled hard against Port, who ultimately seized control of the midfield contest.
"It's not a concern. It's a challenge though," Voss said.
"I'll back the men in that are in there though. They're competitive and they're always looking to get better and the one thing I know they don't like is losing. They don't like losing a contest, let alone four points.
"But we've got to acknowledge we got beaten in there today. Don't say that often.
"And it's not me putting that on them. It's just that that's the fact and that's the story of the game. And so we have to accept the story for what it is and we have to get better."
Carlton's David Cuningham exits the field after sustaining an injury against Port Adelaide. (James Ross/AAP PHOTOS)
Voss brushed off the suggestion Carlton's midfield looked like the "bad old days" when they were over-reliant on Cripps.
"I don't think that just because Patrick Cripps is in good form that that should count against the others," he said.
"He's a great player and he's got his standing in this game for a very good reason. He's captain of our football club. He's had an outstanding season.
"But we know that to be the team we need to be it takes a tribe and we've got to build four, five, six people through that midfield like we do through the defence and like we do up forward.
"So it's not all about Charlie (Curnow). It's not all about 'Weiters' (Jacob Weitering). It's not all about 'Crippa'. So we're gonna build it off some better fundamentals than that."
Harry McKay was a late withdrawal through illness and Voss insisted it wasn't related to his head knock against North Melbourne on Sunday.
McKay was cleared of concussion but the Blues were reprimanded by the AFL for their delay in taking him off the ground.
Despite Carlton saying McKay was merely battling an illness, the AFL on Saturday confirmed they will make enquiries with the Blues in coming days to check how their star forward was managed following the head knock.
"(McKay) had a great week," Voss said.
"He trained, got through all the training sessions, got through everything.
"So no, he was ill and got crook and couldn't play."
Voss had concerns for luckless midfielder David Cuningham, who suffered a dislocated shoulder in the first quarter.
But he played down knocks to Cripps (shoulder) and Weitering (thigh) ahead of a crucial clash with Collingwood on Saturday.
"What we do have in front of us is an unbelievable opportunity and it's well and truly in our control," he said.
"So we've got a month to go and the games become critically important from here."
16°C