Specifically for configuring an analytics extension connection for Tableau Server there is no need to provide a certificate for tsm security vizql-extsvc-ssl enable command and --cert-file parameter is not supported anymore.
The certificate validation works in exactly the same way as it does for Tableau Desktop which means root certificate, self-signed certificate or the whole certificate chain (for Rserve) has to be installed as trusted on Tableau Server nodes.
With this improvement, there is no need to share an analytics extension certificate and it can be validated against what is configured on client machines (nodes running Tableau Server).
The change is for how secure connection is created. When configuring a connection with main menu Help, Settings and Performance, Manage External Service Connection… here’s how new configuration dialog looks like:
What is different there is no link to specify a certificate for secure connection – there’s just Require SSL option.
To compare this is how the same dialog looks in Tableau Desktop 2019.4:
How certificate validation works now is instead of comparing server (TabPy, Rserve, etc.) certificate with user-provided certificate Tableau validates the server certificate and trusts it only if it is installed as a trusted cert on the client OS or if it is signed with a trusted cert.
The server certificate can be signed with an intermediate certificate that can be signed with another certificate and so on until there’s a root certificate Tableau can trust.
For self-signed certificates (those which are not signed by any other certificate) they should be installed as trusted as well.
For how to install a certificate and make it trusted read documentation for your OS (the steps are very different for each OS).
NOTE: Rserve only sends leaf certificate and not the whole chain (there’s an issue opened for that – https://github.com/s-u/Rserve/issues/140) which makes it impossible to validate the certificate on the Tableau side unless the whole chain is installed as trusted.