What versions of Linux do you target?
But we've got customers running everything from RHEL 4.5 to the latest and greatest Ubuntus, Mints and Arch.
All with the same binary installers.
The only linux-related weirdness I can recall in the last two years distro-wise was Scientific Linux, which seemed to have some weird XOrg config issues.
So you integrate with the user's desktop environment, using standard local widgets, theming, integration with UX guidelines, local library dependencies, etc?
Or are you shipping a Qt app with your own dependencies included? If so, Qt is pretty notoriously buggy on OS X (and rightfully disliked by most users as it stands outside all platform conventions), perhaps explaining some of your complaints.
I don't believe there are general UX guidelines for Linuxes, but for things like notification popups, system tray stuff, yeah, we do all that natively to the distro through Qt (DBus handles all that transparently within Qt very nicely).
We ship Qt as shared libs with the binaries and all dependencies we need system-wise statically - i.e. zlib, libpng, libjpeg are statically linked. For things like embedded Python, we have to do the same on OS X anyway, due to different python and zlib versions per OS X version.
As for Qt and OS X regarding nativeness - I keep hearing this, but I never see any good examples. I agree it's possible to create Qt Apps on OS X that looks crap, but it's also possible to make them look native as far as I'm concerned. The only thing I can remember not being able to easily do in Qt regarding OS X widgets is the horizontal grouping of side-by-side buttons in radio-button fashion. But it's easy enough to knock up a QWidget subclass which replicates this. But admittedly I don't think I've tried to replicate every possible OS X control/widget...
I think what might be the most difficult part of making a Qt app on OS X look native is the layout and spacing stuff, which generally does seem to be a bit crap within Qt on OS X.
I can have the same code to do a side-panel with edit values in a panel, and it looks great on Linux and Windows: http://imgur.com/kUI90ry
on OS X: http://imgur.com/yqigBdw
and the spacing and padding is all over the place - so I have to add a manual style-sheet to get the padding and spacing right, which is crap (image above is without extra stylesheet). So I'm not saying it's perfect or easy, but I think it is possible.
on 10.6 (and 10.7), memory allocation/paging is pretty atrocious if you haven't got much free mem left.
Basically, OS X prioritises paging to disk over freeing up Inactive memory (which isn't actually being used for anything at present), which is pretty crap. So it's very easy to make it page when it shouldn't when you allocate loads of memory, and the system will generally just grind to a halt.
Apple fixed this in 10.8, so now it will free up Inactive memory first, before it starts paging.
But neither of these explain any crashes within Nuke.
Does QT basically render a bitmap, or does it use the native controls and just has some odd default styling? If it's the first, then I would probably do the UI in native Objective C/Cocoa and keep a nice cross platform base for the VFX core of the app. Not sure if that's feasible for your application of course.
http://imgur.com/68b9eoO - is native Cocoa moc-up in XCode.
I concede the borders are slightly different, but I personally wouldn't have noticed or minded.
The render button is a custom button - the only difference from the native one is the right-hand menu triangle which isn't native.
Qt has styles, and draws controls itself via vector drawing. And you can customise this yourself: So you've got all the power in the world to do what you want, but at the expense of loads of complexity that no-one's really going to go to the trouble of doing. So basically, it tries to emulate the controls on all platforms, and does a pretty poor job on OS X.
I would side with the reader below, I think that going for a completely different look works better than 'almost' native. Ableton Live is a good example I think that has (almost) the same UI on Mac as on Windows which doesn't look bad on either OS.
I am thinking of Pixelmator, or Apple's Final Cut Pro and Motion for comparison to your screenshot. Even modern versions of Photoshop manage to look much nicer than that.
Qt "emulates" native control appearances by drawing them itself. This means it doesn't get it 100% correct (in OS X's case). You can apply stylesheets to configure all aspects of spacing and appearance, so in theory it's possible to write a stylesheet to completely emulate OS X's controls (or at least the majority of them), but that'd be a lot of work which you probably wouldn't want to do.
I could make it any colour I wanted with stylesheets in Qt - this was a personal app, and I'm not too concerned with what it looks like.