But even if Spicer knew Trump's crowd wasn't larger, Kimmel pressed on, his job as press secretary obliged him to say that it was.
"To get up there and question, on day one, my integrity, I think was not something that I anticipated", he said.
Though he said he has "absolutely not" distanced himself from Trump, Spicer revealed that he does have a specific regret during his tenure.
"There was a faction of people out there that didn't want to give him the credit that he rightly deserved".
Getting serious for a minute, Trump's former staffer went on to say that there many people - especially members of the media - who were constantly trying to undermine the validity of the election.
He also added that he had never seen the President naked when Kimmel asked Spicer about Trump's obsession with size.
After a brief clip from the sketch show was aired, Spicer could be heard chuckling and joking as an aside, "it costed me, like, a lot of money in therapy".
He said: 'Your job is to give him advice but ultimately he's the President'.
Stop using pellet guns in J&K, Amnesty tells govt
He said the pellet guns are only used in Jammu and Kashmir and not at any place in mainland India. Government understands that the damage caused by the pellets is disproportionate".
The company in its announcement said its clients "around the world will benefit from the same candor, wit and insight that Spicer brought to the White House briefing room".
Later during the sit-down, Spicer attributed the crowd size drama as the reason for his "bad start" with the press corp.
Kimmel said, asking Spicer if he tried to talk Trump "out of that line of defense". "And my view was is that if I'm going to have to partner with somebody that I don't believe had the skill set to execute the job, then it was incumbent upon me to either step aside or make my voice known".
"And that's what you sign up to do", Spicer concluded. "When these guys in the press corps go after the president, from the outside it creates a very poor relationship overall and I think there are some areas that could deserve a reset", he said.
Spicer also blamed the media and reporters specifically for "perpetuating myths" increasing tensions. He also explained that he had no personal ill will towards his (brief) successor Anthony Scaramucci, despite thinking he didn't possess the right qualifications for the job.
In reference to the New Yorker article that saw The Mooch unload on Spicer, the former White House Easter Bunny said he would not 'relish in someone else's problems, ' but that it 'proved my point'.
He ended the late night appearance by insisting that Trump does, in fact, want to be President.