Although there is evidence that previous buildings dating back to 1184 existed on the island, the remains of the castle that stands there today was built in the 1800's. Many poems were written about the families and wanderers that occupied the island including the Hag of Lough Key...
There is reference to Castle Island in the annals of Lough Ce as early as 1184. During this time the park was called Moylurg and the Kings of Moylurg were the McDermotts.
The McDermott’s official residence was on The Rock, now called Castle Island. As space was limited on this small island they had another residence on the mainland where the Moylurg Tower stands today. The McDermotts ruled this area until the 17th century when it was granted to the King family from England under the Cromwellian settlement. The King family spent their time between the town of Boyle and Moylurg which they renamed Rockingham. One of the large mansions they built was called Rockingham House and was built where the Moylurg Tower now stands.