This is an old question but I will give my perspective on it for future reference.
First of all, let's get some basic certificate knowledge up on the table so we can agree on what we are discussing. # Server Certificate A webserver certificate (for normal TLS, regardless of m for mutual or not) is a certificate that is used on the web server side to verify to clients (browsers mostly) that the server owner also owns the domain (this is verified using established certificate signing and verification techniques), verifying that the certificate is valid is done during the TLS handshake, the browser checks the computer's trust store for the certificate that was used to sign the webserver certificate to see that it is valid. If it's valid, handshake continues. # mTLS Using mTLS (mutual TLS), the client also has a certificate that it presents to the server for the server to validate against a trust list. The trust list is normally either a file of concatenated root- and intermediate certificates that the web server uses for verification, or the server operating system's trust store. If the intermediate certificate that has signed the client's certificate is in the trust store, it is valid. ## CRL Both of these techniques also uses Certificate Revocation Lists, which are referenced inside the information of the certificate itself and it's chain parents (intermediate, root) to see if the certificate was revoked after being enrolled. If it was revoked, the handshake will fail and the connection terminates.
These handshake dialogues will allow any client who has a signed certificate from a CA (Certificate Authority) that the server trusts (inferring that the handshake goes through the above steps successfully). Thus, using this in a completely public domain, there must be some way to determine which clients to register for login and which to deny. # Public domain (internet) If you are going to use this kind of solution (mTLS) in a completely public environment (internet), with *any* user, then I don't think certificates are the correct way of authenticating or authorizing users, without some other way to enroll them as users or some other way to authorize their behavior after authentication. # Custom limited domain (inside organization or between select few orgs) If it is used in a more limited domain, say inside a company or between organizations that operate in the same field, it is not uncommon to have custom CA's either completely internal to the company or a CA organization that only signs certificates along a standardized procedure between a limited number of companies in the same field. The CA is then pushing the responsibility of this process that it is legit, to the companies.
Using mTLS in such a situation or domain is much less hassle, probably, you can then only trust the intermediate or root cert of that custom CA on the serverside and then allow connections from clients with a custom CA certificate that you trust.
[Public Key Infrastructure (Wikipedia)][1] # Authentication vs Authorization That said, there is a difference between authentication (letting a user in "through the gates") vs authorization (what a user is allowed to do once inside). [Authorization vs Authentication (Okta)][2] I think this is at the heart of your question. You could accomplish authentication by letting anyone with a client cert in, authenticating them as a distinct user the second time they log in, by validating the thumb print.
# Certificate expiration The problem of saving the thumb print is that certificates have an expiration date and normally, when enrolling a fresh certificate, the thumb print and serial will also be rerendered, so saving that in the database is not really a feasible way of keeping track of users over time. Inside a Custom CA and the policies around how certificates are handled, the subject field of the certificate can contain a user identifier that, together with the certificate chain of CA's would then guarantee that a certificate signed by a distinct authority with a distinct subject field will be the same user entity regardless of certificate renewal and thumbprint + serial rerendering.

[1]: https://en.wikipedia.org/wiki/Public_key_infrastructure [2]: https://www.okta.com/identity-101/authentication-vs-authorization/

I agree with EJP's answer, but since you've accepted a different one, here are further details on what the issue with self-signed certificates are going to be.
Using a self-signed certificate on your server can be done. You'll just need to configure all the potential clients to use trust this certificate. This can be done by importing the certificate (without the private key) into each machine's trusted store, or within the browser's trusted certificate repository (for example, Firefox uses its own, independent of the OS on which it's running). Self-signed server certificates here are mainly a special case of a custom CA certificate, where each client needs to know about each server it may use (with the downside that self-signed certificate can't be revoked). This is OK, but this can quickly become a problem if you have than a few machines, in practical terms.
Your main problem will be about self-signed certificates as *client* certificates.
Client-certificate authentication is initiated by the server, which sends a TLS Certificate Request message to the client. This message contains a list of names of Certification Authorities which it's willing to accept. Clients then use this list to select which certificate to send: they look for certificates (for which they have the private key) that have an Issuer DN matching one of the names in this CA list (they can also use certificate chains, if intermediate CA certificates are required to make the link to an Issuer DN that's in this CA list).
Most clients will simply not send any client certificate at all if they can't find a certificate that matches these conditions. This is going to be your biggest problem when trying to use self-signed client certificates.
A way around this problem is to get the server to send an empty list. This gives more freedom to the client as to which certificate to choose (it's then up to the server to choose whether or not to accept it, but that's would always be the case anyway). This is explicitly allowed by TLS 1.1 and previous versions (SSLv3/TLS 1.0) are silent on this topic, although in practice, it works fine with most SSLv3/TLS 1.0 stacks too as far as I know. (This can still lead to clunky interactions when automatic certificate selection is done by the browser, for example, since it makes it difficult to make that automatic selection correctly.)
Java's `X509TrustManager` can be customised using `getAcceptedIssuers()` to return an empty CA list in the TLS Certificate Request message. As far as I know, in C#/.Net, `SslStream`'s `RemoteCertificateValidationCallback` doesn't have its equivalent. As far as I know, this list is built from the machine's trusted certificates list when using IIS/`SslStream`. (Note that this is independent of the certificate verification itself, it's just about the server advertising which certificates it might accept.)
In addition, if you want to use client-certificate authentication with self-signed certificates, you'll have to implement your own verification callbacks in the server to perform this authentication. A simple example would be to extract the public key from the certificate you get, compare it against a pre-defined list the server has, and then use the user name you have in that pre-defined list. You just can't trust anything a self-signed certificate says unless you have an explicit way of checking its content against something you know. Implementing a callback that just says `return true;` will not perform any authentication all all. Anyone could build a certificate with the same names or attributes and different keys. (It would almost be better to trust each client-cert explicitly in your server trusted certificates store, although this may lead to long CA lists in the TLS Certificate Request message.)
If you don't want to rely on a commercial CA, build you own CA. It can be a bit of work (mostly administrative), but at least you'll have a cleaner resulting system. By configuring your clients and your servers with that CA certificate, you should be able to expand to more clients and servers without changes there, and you'll also be able to benefit from the normal CA list behaviour when the client certificate is requested.