According to the TS Setup documentation, you can change your TYPO3 site's doctype by adding the following to your TS Setup:
# Change TYPO3 doctype on normal installation config.doctype = xhtml_strict
However, if you are using the new Templavoila Framework, this value is overriden by a page-specific setting. So, if you want to change the doctype and you are using the Framework, you must instead use the following lines:
# Change doctype when using TV Framework page.config.doctype = xhml_strict
where xhtml_strict can again be interchanged with any of the valid values listed in the config documentation above.