Officers said the suspect, a white, bald man in his 40s, approached the girl from behind at about 2.40pm on Tuesday by the King's Statue near the beach in Weymouth.
He lifted up the girl's top and touched her inappropriately on the bottom, police said. The man carried on walking towards the Jubilee Clock Tower, about five minutes away on foot.
The suspect was wearing a blue T-shirt and pushing either a pram or a wheelchair, police added.
Officers are combing through CCTV footage to get more information about what happened, and are also calling for witnesses to come forward.
Weymouth Police Constable Toby Norman said: “This incident understandably left the victim feeling very distressed and we are determined to identify the man responsible.
“We are carrying out a number of enquiries, including a trawl of CCTV footage in the area, and I am urging anyone who witnessed what happened to please come forward.
“I am particularly keen to speak to a group of three older women who I understand were walking near to the man at the time of the incident and are likely to have seen what took place.”
Anyone with information is asked to contact Dorset Police at www.dorset.police.uk, via email at firstname.lastname@example.org or by calling 101, quoting occurrence number 55200115169.
Alternatively contact the independent charity Crimestoppers anonymously online at Crimestoppers-uk.org or call freephone 0800 555 111.