Skelmorlie Castle was formerly a seat and stronghold of the Clan Montgomery. During the reign of Robert III (1390–1406) the lands of Skelmorlie were held by the Cunninghames of Kilmaurs; in about 1460 the northern portion passed into the hands of the Montgomeries as Skelmorlie-Montgomerie, the remainder continuing as Skelmorlie-Cunninghame. Anne, sister of Alexander de Montgomerie married a Cunninghame of Kilmaurs and this may explain the division of the lands.

On 6 June 1461 Sir Alexander de Montgomerie of Ardrossan, the first Lord Montgomerie, granted the lands of Skelmorlie to his second son by Margaret Boyd (daughter of Sir Thomas Boyd of Kilmarnock), George, thereby was the founder of the cadet branch, the Montgomeries of Skelmorlie. Another view is that Skelmorlie Castle was built by a brother of Hugh, the first Earl of Eglinton. This Hugh was the son of the second Lord Montgomerie and the grandson of the first Lord Montgomerie. John was the second laird and he married Marion Dalzel, by whom he had a son, Cuthbert, the third laird. Cuthbert married Elizabeth, daughter of Patrick Houston of that Ilk and their eldest son George became the fourth laird. George married Lady Catherine Montgomerie, youngest daughter of Hugh, first Earl of Eglinton. They had a large family and the eldest, Thomas, became the fifth laird. Thomas died in 1566 and his brother Robert inherited Skelmorlie.

Robert Montgomerie, seventh laird, was knighted by James VI and in 1628 created a baronet by Charles I. He was the son of another Robert; his mother was Dorothy, daughter of Lord Sempill. This Sir Robert married Margaret, daughter of Sir William Douglas of Drumlanrig Castle. The sixth laird, Robert, murdered Alexander Cunninghame, commendator of Kilwinning Abbey at his castle and palace of Montgreenan in 1586, following the murder, by the Cunninghames, of the fourth Earl of Eglinton. The elder Sir Robert and his eldest son where in revenge killed by Patrick Newall of Newark Castle.

Sir Robert, the seventh laird and first baronet, was one day discovered by Patrick Newall in Newark Castle; presumably also bent on revenge for his father and brother. Patrick is recorded as saying "Come down Robin, out o' that corner, come down, man, to me, who did you so good a turn as to make you young laird and auld laird o' Skelmorlie in one day." Patrick and Sir Robert in fact became friends. Sir Robert married Margaret, daughter of Sir William Douglas of Drumlanrig. She was a famous beauty and the subject of ballads by Alexander Montgomerie of Hessilhead. The third baronet and ninth laird, Sir Robert, married Antonia, co-heiress of Sir James Scott of Rossie in Fife, by whom he had four sons and four daughters. This lady was an ardent covenanter and her husband was repeatedly fined for her attendance at conventicles. Robert died in 1685 and was succeeded by his son James. They had seven sons and four daughters.

Sir James, fourth baronet and tenth laird, was a member of the 1689 Parliament and Convention, like his father Robert. It is recorded that his mother wrote in 1684, severely chastising him for not providing for her and her four fatherless children. He was one of those chosen to administer the oath (the offer of the Crown) to King William and Queen Mary in London. He was dissatisfied with his reward of Lord Justice-Clerk and become a supporter of the deposed King James II, joining him at St Germains in France; despite this support Sir James, a protestant, was not fully welcome by the Catholic King James and despite a promise to create him Earl of Ayr, he died, of 'vexation', in September 1694 in London. He had married Lady Margaret Johnston, daughter of a co-conspirator, the Earl of Annandale They had two sons, Robert the elder and William.

The Lord Provost of Glasgow, Hugh Montgomerie of Busbie, purchased the estate from the fourth baronet; taking possession in 1731 and inheriting the title as 6th baronet and eleventh laird. In 1687 he married Lilias Gemmel, daughter of Peter Gemmel, merchant in Glasgow and Christian Boyd his wife. Hugh was one of the commissioners appointed to negogiate the union with England, serving as a member of Scotland's last parliament. He was appointed to the first parliament of the United Kingdom as the representative for Glasgow. Hugh died in 1735 without issue and being the last direct desendent, the title became extinct.

The Skelmorlie estate passed to the 4th baron's daughter, Lilias, who married Alexander Montgomerie of Coilsfield. Lilias's son Hugh became the 12th Earl of Eglinton and his brother held the property of Annick Lodge. Lilias had five sons and three daughters and died in 1783, a few weeks before her husband. She sold her lands in Renfrewshire and purchased lands contiguous with Skelmorlie.

The original Skelmorlie Castle may have stood on a motte which is situated in a defensive position on a promontory on ground lying between a small burn and the Skelmorlie Water. The oldest portion of the present day castle was built in 1502 and added to in 1636, when corbelling and parapets were removed, whilst angle turrets, later roofed, were added. Timothy Pont in his survey of Cunninghame around the year 1600 describes Skelmorlie Castle as "a fair well built house. It is certainly, in point of situation, a very pleasant and most delightful place, with its old-fashioned gardens, terrace and shrubbery. The view from it over the Firth of Clyde, to the opposite islands of Bute, Arran and Cumbraes, is not to be surpassed in picturesque scenery by any prospect in Britain." The baronial hall was on the first floor and as a more modern day dining room it had the Montgomerie coat of arms on the roof with the date 1762. The castle chapel, converted in the 19th century to stables and servants' quarters, still stands at the western side of the southern courtyard. Sir Robert's 17th range survives in part.

The grounds in the early 19th century are described as "beautifully laid out, and covered with rich and luxuriant woods. The principal disadvantage of this fine old mansion, as a modern residence, is certainly the difficulty of ascent to it, from the steepness of the hill on which it stands; but otherwise an agreeable, nay, delightful summer retreat."In the 1855 OS survey a bowling green and a sundial, possibly a Scottish sundial, are shown. The old baronial mill still exists in the name 'Milnburn' close to the Skelmorlie Water and Skelmorlie Mains is marked.

The silhouette of the keep has remained largely unchanged for over five centuries. This red sandstone rubble construction was rough-cast harled in the early 1960s following a fire in 1959, which burnt out the upper floors of the keep and of the west wing. The wing was demolished and the keep restored; the architects were Noad and Wallace of Glasgow. Major-General James Montgomerie, of Wrighthill, M.P. for Ayrshire lived at the castle for a long period in the early 19th century, being the brother of the 12th earl and grand-uncle to the 13th earl. During this time the castle remained as a little altered, but run down example of a tower castle.

The castle was tenanted for many years (1852–74) by the textile merchant, James N. Graham of Glasgow, a patron of arts with an outstanding art collection, who restored the old tower and greatly extended with the permission of the Earl of Eglinton. His architect was William Railton of Kilmarnock. The 16th Earl of Eglinton moved here from the abandoned Eglinton Castle Montgomerie family seat in the mid 1920s, but it was sold by the 18th Earl of Eglinton, having been leased to the Wilson family since 1956. The 16th Earl died at Skelmorlie Castle in 1945. The castle became the property of the Wilson family in the mid 1970s.