You can customize logging by creating a file named ".tc.custom.log4j.properties" either in your home directory ("user.home") or the working dir of your java process ("user.dir"). In there you can do the usual log4j configuration. This is supported for versions terracotta-3.6.x and above.
Other than that you can use the following properties (applicable for client and server)
logging.maxBackups = 20 [The maximum number of backup log files to keep]
logging.maxLogFileSize = 512 [By Default in MB, no need to mention M, once file size reached, it will rolled over]
I am using terracotta-3.6.2 and tried your advice regarding the custom log4j file. But Terracotta continues to write in "terracotta-server.log". Can you please elaborate on how to setup the custom logging.