We recently had an issue in an environment that I am looking after that caused the VPXD.exe (vcenter server service) to use 100% of the memory allocated to a machine, plus all the disk space on the server (as the page file was set to system managed and just kept growing) until the server stopped responding. After much investigation, the following lines were noted in the vCenter server log (appearing a lot). The error is as follows:
[05444 error 'Default'] SSL StreamImpl::DoServerHandshake (0000000028a34600) SSL_accept failed with Unexpected EOF
[05444 warning 'ProxySvc'] SSL Handshake failed for stream TCPStreamWin32(socket=TCP(fd=5470) local=<vcenter IP>, peer=<remote client connection>), error=SSL Exception: Unexpected EOF
That error message basically translates to a VI Client instance connecting in to vCenter, when this occurs, each connection consumes additional memory which isn't released. In our case, we had an incident where the VM Tools instance was ripped out of a management server, this management server used the VMXNET 3 driver. In its unusual state it was polling the vCenter server a large number of times a minute (like it was flapping). Once VMware Tools was reinstalled on the management server, vCenter calmed down and returned back to a normal amount of memory usage.