The 56-year-old, his girlfriend Lauren Silverman and their baby Eric were in bed when the raider sneaked in.
The thief hunted for valuables and stumbled across the key to Cowell’s safe.
He then cleaned out all the cash and jewellery inside, according to a source.
“Unfortunately security was anything but safe,” said the source.
“The burglar found the key and bingo, he was in.
“Some of the gems were sentimental and there was some cash in the safe.”
The value of what was taken could be well into six figures.
The alarm activated at 2.20am yesterday morning and awoke Cowell’s bodyguard.
But by the time he reacted the burglar had fled from the house in Holland Park, west London.
“Obviously Simon and Lauren are really badly shaken by this,” a family friend told The Sun.
“It is a horrible violation of their privacy. But more than anything they are incredibly relieved no one was hurt.
“As upsetting as it is, Simon can cope with losing possessions, as long as his family are safe. That is his one and only concern.
“Still, it’s incredibly worrying and he is obviously going to review security.
“He likes to live as normal a life as possible and doesn’t want his house to be like Fort Knox, but his family need to feel safe in their own home.”
The TV star has a personal bodyguard on duty around the clock as well as ho-tech alarm systems.
It was unclear if the burglar specifically targeted the mansion or if it was a random break in.
Cowell tried to hide his anguish as he attended rehearsals yesterday ahead of tonight’s X Factor semi-final.
“This could not have come at a worse time,” added the source.
“Simon is preparing for one of the biggest weekends of his year and needs to be completely focussed.
“Obviously he wants to be with his family but he knows they are safe and are being well looked after, so he needs to be professional and get on with the job at hand.”
Cop cars were parked outside the address following the break in as forensic officers examined the scene.
They were also trawling through CCTV cameras to check for any clues to the identity of the suspect.
A Met police spokeswoman said: “Police were called to an address in W14 on Friday 5 December approximately 02.20hrs to reports of a burglary in progress.
“Officers attended the scene but the suspect had left the area.”
This was the second time Cowell’s property has been targeted.
In 2012 stalker Leanne Zaloumis broke in by smashing a window with a brick.
She was found lying on the top shelf of a walk-in wardrobe and given a community order for breaking in.
Source: Daily Star, UK
Follow us on Twitter at @thesignalng
Copyright 2015 SIGNAL. Permission to use portions of this article is granted provided appropriate credits are given to www.signalng.com and other relevant sources.