Tried it on my android phone, it's cool for typing and such, but for just generally going through the menus, the controller is faster. So i'd definitely use it if i'm using the browser or any form of typing, saves me money on buying that little keyboard accessory.
So i suppose it depends on what you use your xbox for. I haven't done any sort of gaming that uses the smartglass stuff (i.e. extra information and having a secondary screen) , but i don't foresee myself using that instead of the controller for gaming. I also do alot of videos and music streaming from a library on my PC, controller probably still faster there too.
The new features that came with the dashboard, like Internet Explorer and such, also need an Xbox Live Gold subscription, so if you're not exactly hardcore gaming you might not be able to use these things anyway, which makes the Smartglass App a glorified menu-navigator.