You can’t access anonymous nodes that way from JS, you have to get creative with what the XUL binding is exposing to you. Using DOM inspector to inspect XBL and the JavaScript object is useful here, as well as just using the javaScript shell to delve down into objects.
Here’s what you want:
var selectedTab = ko.views.manager.currentView.parentView.tabbox.selectedTab;
selectedTab.setAttribute("style", "background: #ff0000");
I’m not following your logic there, that looks like a really convoluted CSS rule. You dont need to do any trickery to access anonymous elements through CSS, only when accessing them through JS.