Setting up environment
First of all you have to make sure, that your PHP server is able to conect to Microsoft SQL server at all. This requires a PHP extension named SQLSRV which you can download from the Microsoft's official website and unfortunately only works on newer versions of Windows OS.
You will have to add a line to your php.ini file to tell your PHP script to use the extension. Depepending on your PHP installation you have to use one of the four extension files: php_sqlsrv_53_ts.dll, php_sqlsrv_53_nts.dll, php_sqlsrv_54_ts.dll or php_sqlsrv_54_nts.dll, where ts stands for Thread Safe and nts for Not Thread Safe. If you are not sure which one to use you can go to your Joomla Adninistration > Site > System Information > PHP Information and check your PHP version and a parameter called Thread Safety. Finally you open your php.ini file and insert a new line:
extension=php_sqlsrv_54_ts.dll
In the next step you will have to install a software called Microsoft SQL Server 2012 Native Client, to make the connection work. This is also freely available from Microsoft's website or you can use the download links on the official PHP website.
Now restart your PHP server and you should be good to go.
If you use a shared hosting, you should get in contact with your hosting provider, to set up environment for you.
Connection setting for Tools JX
To connect to a another database than the one thar Joomla uses, you have to select "Other" connection by using the radio button in the "Connection" area. Next, you fill in the connection data. To connect to MSSQL database using SQLSRV use the Database Type "sqlsrv". Other settings are straightforward. Fill in the host address (domain or IP address), database User and Password and Database Name.
On the end hit the "Check and List Tables" button, to test the connection. If ewerything is set properly a dropdown box should appear on the right with all the tables listed. Now you can proceed with selecting the table and columns.