In either case, the VM uses VMware Tools to identify the state of the VM and its application and then reboot the VM if necessary.
What is the difference between VM Monitoring Only and VM and Application Monitoring?
VM Monitoring Only utilizes VMware Tools to check regularly for heartbeats and I/O activity from the VMware Tools process. Depending on the Monitoring Sensitivity setting, vSphere HA will know how often to look for heartbeats and how often the VM should reboot.
VM and Application Monitoring also utilizes VMware Tools and can monitor the VM just like the VM Monitoring Only setting but it can also monitor the application, running on the VM. In order to enable Application Monitoring the application has to support VMware Application Monitoring or use the appropriate SDK.
At a minimum it is best practice to enable VM Monitoring Only to ensure maximum availability for your most important VMs. Note, this setting can be adjusted per VM via VM Overrides in the Cluster Settings