The incident happened around 10 pm on Friday May 8 when the 21-year-old victim was walking along the town’s Church Street.
Police say a man approached the woman from behind and pulled her into a doorway where he subjected her to a sexual assault.
The young woman managed to fight the attacker off and contacted police.
A spokesman for Police Scotland described the suspect as white, bald, between 30- 40 years old, around 5 ft 7 inches in height, with a medium build and stubble.
He was also said to be wearing a royal blue hooded top (with the hood down) which had white tassels attached and blue denim jeans.
Officers are reviewing CCTV images from the neighbourhood in an effort to glean more detail about the suspect.
And they are appealing for up to five potential witnesses to contact them urgently.
The police spokesman said: “Enquiries so far have revealed that a group of four people and a person walking a dog were going along King Street at the time of the incident.
“The four were carrying umbrellas and crossed the road at the junction of King Street and Church Street and continued along King Street. The dog walker also continued along King Street.”
Enquiry officer, Detective Sergeant Stephen Bell, said: “A young woman has been left extremely distressed by this ordeal, she was very brave and fought her attacker off but it is imperative we trace the man responsible.
“I would appeal, particularly, to the five people who were seen in the area at that time and to anyone else who was in the vicinity to get in touch with us.
“It’s possible you saw the suspect and thought nothing of it at the time, but if you were in the area or you have any knowledge of the suspect or crime, please do contact us.”
Dumfries CID can be contacted through telephone number 101 or information can be passed anonymously to Crimestoppers on 0800 555 111.