Databricks: Using Python logging module in notebooks

One common problem many developers face when using the Python logging library in Databricks Notebooks is that the log file does not get stored in the Workspace or cloud file storage. This script addresses that problem.

Sample Log Monitor

This script uses TimedRotatingFileHandler, so the log files are rolled every 60 minutes, and manage 5 backup files.

file_handler = TimedRotatingFileHandler(log_file, when="m", interval=60, backupCount=5)

This script displays the output on the screen and stores it in a log file. If you don’t want to see it on your screen, remove or comment out the following lines.

console_handler = logging.StreamHandler()
console_handler.setFormatter(log_formatter)
console_handler.setLevel(logging.ERROR)
logger.addHandler(console_handler)

The code that enforces the creation of log files in Databricks Workspace or cloud storage is this: It's always a good practice to close the opened file handler, but many developers forget to do that.

file_handler.close()

Complete Script

import os, logging
from logging.handlers import TimedRotatingFileHandler
from datetime import datetime

def setup_logging(log_dir):

    #Create a new folder, if folder already exists don't throw error…