Caravans and a range of vehicles began taking up spaces in the Murray Street car park on Monday evening, with numbers increasing as the week went on.
A Dumfries and Galloway Council spokesman said: “Our council is aware of the caravans in the car park and has visited to ensure no anti-social behaviour is taking place.
“Their presence will continue to be monitored and the group have indicated their intention to move on in the near future.”
He added: “Advice has been provided regarding vacancies on the council-operated traveller sites. If there are reports of anti-social behaviour related to this unauthorised encampment, the council has multi-agency arrangements in place to respond.”
Regular users of the car park expressed frustration when they were unable to find a space.
One irate shopper hit out on Wednesday: “There are few enough parking spaces in Annan town centre. I’m afraid I’ll be doing my shopping elesewhere for the time being. If the travellers have proved anything it is there is a lack of adequate parking in this town.”