The phrase "an unusual system" is correct and usable in written English.
You can use it to describe a system that is not typical or deviates from the norm in some way.
Example: "The researchers discovered an unusual system of communication among the species that had never been documented before."
Alternatives: "a unique system" or "an atypical system".
They also introduced an unusual system.
This is an unusual system, in that it is worked by just one muscle.
In addition, Mr. Turner established an unusual system of bureaucratic competition.
These contradictions and abrupt reversals, which caused considerable puzzlement when they occurred, were to be explained partly by an unusual system of speechwriting that Mr. Nixon had adopted.
In the video Attenborough describes how the sloth has developed an unusual system of camouflage by letting its fur become infused with algae.
That project, 45 miles west of Phoenix, will use an unusual system to convert the electricity into alternating current, the kind that is transmitted on the power grid.
On the blog onigame.livejournal.com, the puzzle master Wei-Hwa Huang (who came in third at the world competition in March) devised a unusual system, using colored loops, for solving sudoku that is more knotty than seems possible given the puzzle itself.
The IL-6 receptor system has an unusual organization.
Council members are elected from various wards under a highly unusual system in which candidates stand as independents rather than party representatives.
Instead, a highly unusual system was devised – which lies at the centre of the battle now being waged between the committee and the CIA.
The news of the potential investigation comes after analysis of a similarly unusual system - the process used to determine the London interbank offered rate, known as Libor – uncovered manipulation and triggered multi-billion dollar fines against a group of banks.
