James Dunsmuir called his new town at Oyster Bay 'Ladysmith' to commemorate the raising of the siege of the South African town of the same name during the Boer War. Several streets also bear the names of British generals, including Baden-Powell. The original Lady Smith was the Spanish wife of Sir Harry, one-time governor of the colony.

The Chemainus First Nation includes Ladysmith Harbour in its traditional territory and the largest of its three populatrion centres today is situated on the forested point of land across the harbour from the Town of Ladysmith. Lack of good farm land discouraged European settlement and the heavily timbered slopes rising steeply from the shoreline made logging difficult. The situation began to change in 1898 when James Dunsmuir began to ship coal, supplied by his new mines at Extension, a few miles away, from the deep water loading wharves at Oyster Bay.

By 1901, Ladysmith was "a hustling, bustling town" with "some 700 men and boys" taking the miners' train to the Extension mine. By 1904, there was a population of 2,500 and 13 hotels. In 1902, the Tyee Copper Company on Mt. Sicker built a three and a half mile aerial tramway from the mine to the Esquimalt and Nanaimo Railway at Stratford's Crossing. From there the ore was brought to Ladysmith where the company built a smelter and loading wharf. The Town of Ladysmith was incorporated in 1904.

The coal mines closed in 1931, and half the population moved away. But in 1936, heavy winds blew down acres of trees to the west of the town, and Rockerfeller interests, who owned the land, sold the lumber to Comox Logging & Railway Company. By 1948, 750 men were working in the woods and the company headquarters were moved to Ladysmith, which again prospered.