"with ceremony" is a correct and usable phrase in written English.
It is typically used to suggest a formal, traditional act or event. For example, "The couple wed with ceremony in a grand church."
Turkeys, indigenous to Central America, were once ceremonial offerings and this mole-like concoction is served with ceremony.
He was buried with ceremony in Westminster Abbey close by the grave of Geoffrey Chaucer.
The Security Council sits in a chamber replete with ceremony and diplomatic niceties.
Respect to do with ceremony – what happens when you visit the House of Lords.
She is a team captain for reasons that have little to do with ceremony or sympathy.
Certainly the revised date for the end came with ceremony, and finality.
Others went ahead with ceremonies at the church.
"It is supposed to be a national day of remembrance, with ceremonies across this country.
Bollywood award season is in overdrive with ceremonies being held in quick succession.
Cars are scheduled to finish their run Saturday, with ceremonies concluding Sunday.
Six years later, the Sons proposed that the day be celebrated with ceremonies in every state.
