To abort in a basic msi project through a custom action, one needs to return ERROR_INSTALL_FAILURE in the installer code.
The custom actions which gets executed successfully should return the value ERROR_SUCCESS.
The custom actions which gets executed successfully should return the value ERROR_SUCCESS.