As discussed on the Tools page, MKVtools is an Objective-C application that acts as an interface for a number of UNIX commandline applications. So if you encounter problems with MKVtools, chances are that it's an issue with one of the UNIX applications.
To help evaluate causes of strange behavior, you can select to keep the log files that are created during processing. This option is found in the MKVtools preferences . Examining these log files should give you some hints regarding what the Unix applications are having issues with. The web sites for the individual Tools could also prove helpful.
Probably the best resource would be useful Mac video forums found here or here. You'll find lots of people there willing to help and with more knowledge about general video issues than I have. Also, I periodically read this forum and can chime in with specific MKVtools issues.