The primary factor for the decline of the Vijayanagara empire was the Battle of Talikota in 1565, where an alliance of Deccan Sultanates, including Bijapur, Ahmadnagar, and Golconda, defeated the Vijayanagara forces. This alliance, coupled with internal conflicts within the empire, led to its downfall.