I have managed to do a bit better. Instead of just over the threshold, I have moved up to an rmse score of .9308. This had 21 features, climbing one feature at a time -- with an eye on probe rmse to prevent too much over training.

Back to the think pad now to see what else to try.