This is a great question!!!
You raise some key points that have been at the centre of many debates over breed standards -- the show standards established by kennel clubs for showing dogs. The answer to whether colours 'matter' is in the broadest sense, 'no' except in an aesthetic sense for breed standards, yet that comes with a strong 'but...'.
There are colours and breedings that genetically do run risks -- sometimes, genetically-carried health issues are linked to coat colours. For example
merle colours can carry serious health risks, and my understanding is that all-white, and chocolate (brown) cavaliers can also have coat-colour related health issues. Some breeders have begun to introduce merle coloured cavaliers
which really seems the last thing the breed needs, given it already has numerous health challenges and a problematically narrow gene pool, so throwing merle problems in there is kind of
.
But that last point -- increasingly narrow genetic diversity -- is one possible argument for broadening the breed standard in terms of colour. Yet while some breeders claim to be working to reintroduce genetic diversity by breeding chocolate or black cavaliers, they are usually just creating further narrow branches as these colours were rare, and always were pretty rare. And keep in mind this is a reconstructed breed so started with a narrow gene pool and sites/breeders that state the breed standard was 'only' established in the 40s and before that, there were all sorts of colours, are not really truthful. The CKCS breed was only recognised when the CKCS breed standards were set les than 100 years ago, after a couple of decades of trying to recreate a longer nosed small spaniel of the type popular in Charles II's reign (they were usurped by breeding for flatter faces after William of Orange came to the throne and favoured flatter faced breeds, so they became fashionable). It took several decades to get a longer nosed spaniel that bred true and became the CKCS. Some longer nosed small spaniels were still around, and can be seen in Victorian paintings, for example, but weren't as common as flatter-faced King Charles (aka English Toy) spaniels.
The whole idea of breed standards came from the Victorians as did the concept of the modern dog show. It makes sense to have some sort of breed standard or judges have nothing on which to judge a breed type. Colours are just one part of that and in reality can be interpreted with some flexibility. So it isn't uncommon to see dogs that are technically 'mismarked' still winning in shows. Before standards, 'breeds' were a fairly loose concept and 'breeds' could be outcrossed to other 'breeds' to bring in desired characteristics.
The big problem now with standards is, as you note, health. Gene pools have narrowed and it's increasingly recognised that breeding for certain physical characteristics also can have significant health consequences, as can lack of genetic diversity. So pretty much all purebreds have some breed- specific health problems.
But dropping faults based on markings is probably is not really going to address this in any significant way, I think, though it could indeed get a few more dogs into breeding problems by show breeders. But the vast majority of cavalier puppies are bred for a mass market by breeders or puppy farms that don't care at all about mismarkings or dogs that really veer way off the breed 'type' too, much less caring at all about health and testing, so colour is not influencing breeding choices much in the broader pet dog market -- and of course 'mismarkings' don't matter at all with pets that aren't being shown, anyway. There are current arguments that careful, science-based outcrossing to other breeds is needed to bring back in some genetic diversity but this is very different from so-called designer crossbreeds, or just random outcrossing, as many of the breeds chosen (like all of those seen in the designer crosses -- poodles, bichons, cockers etc) have the same or other serious health issues so can just cause even more problems.
It summary: it's complicated!!! That's my take but someone closer to show breeding might have good insights and the health issue is a big and nuanced one too.