Even before the new church of St Francis Xavier was consecrated in 1735, Vesela Gora was already a celebrated pilgrimage destination. Chapels marking the Way of the Cross were erected on the slope of the hill in 1730. Every year, people formed processions for the Way of the Cross and for pilgrimages. As well as pilgrims from all the towns in Dolenjska and a large part of Štajerska, the pilgrimages attracted people from Bela Krajina, Croatia and Gorenjska. Pilgrims on their way to Kum would also stop at Vesela Gora.
Pilgrims would often stay in Vesela Gora overnight. The more distinguished would spend the night at the priest's house. Others would sleep in the pilgrim house with its tavern, in houses in the village, in the open air or even in the church. In 1751 the Bishop of Görz ordered a ban on overnight stays in Vesela Gora in order to prevent opportunities for sin and indecency. The result was a fall in the number of pilgrims: only those who were able to reach the church and return home the same day continued to come. In 1761 a large new priest's house was built. In 1782 Emperor Joseph II suppressed pilgrimages, brotherhoods and special devotions. Up until that time, the priests at Vesela Gora had not come under the administration of Šentrupert but were directly under the authority of the Provosts of Novo Mesto and the Bishops of Pedena (today Pićan, in Croatian Istria). They also had special privileges as regards indulgences. They supported the church financially and also devoted one share to the poor and the sick. After 1801, however, it was priests from Šentrupert who celebrated Mass in Vesela Gora.
The great assemblies of pilgrims, processions and ceremonies, like those for Whitsuntide, St Margaret's Day, the Nativity of Mary, and the Feast of the Guardian Angels, were no more. Despite the ban, pilgrimages revived over time, but on a smaller scale. In part this was influenced by the vicinity of other pilgrim churches. People felt more at home venerating the Virgin Mary and certain other saints than St Francis Xavier. In the 19th century a statue of Our Lady of Lourdes was brought to the church in Vesela Gora. Rather than to St Francis Xavier, people commended themselves to her, to St Anthony of Padua, to their guardian angels and, particularly in the case of mothers, to St Anne. Nowadays very few people in Vesela Gora are aware that St Francis Xavier was once considered a powerful intercessor at the hour of death and that people prayed to him for a strong faith, to grant them sufficient daily bread and to protect against cattle plague and storms.
Pilgrimages were profitable for the people of Vesela Gora. The church was well maintained, since the collection box was quite full during solemnities, even though many pilgrims were poor. The church was also supported by the counts who resided in the manor house up until the Second World War. After their departure, the manor house was turned into an inn.
During pilgrimages, almost every house (except the sexton's) became a tavern or wine shop, serving (untaxed) homemade wine. The local constables turned a blind eye. In some places they also offered hot meals to pilgrims. The richest inn would slaughter several pigs and calves and sell the meat during pilgrimages and fairs. Local children – shepherd boys – would hold out their hats to the pilgrims as they passed. In this way they might earn the odd copper coin and, from time to time, a sweetmeat bought by the pilgrims from one of the roadside stalls.