Articles

Details: By J. Smith; November 23,2024; Last Updated: 21 December 2025; Hits: 889

# Import packages

# For data manipulation
import numpy as np
import pandas as pd

# for displaying and modifying the working directory
import os as os

# For working with datetime objects
from datetime import datetime

Abstract

Managing continuous glucose monitor (CGM) data efficiently becomes increasingly challenging as datasets grow larger. This post details a streamlined approach for preparing a base dataset to facilitate CGM data analysis. Key steps include optimizing the data download process, filtering relevant information, formatting and enriching the dataset, and preparing it for visualization. By implementing these methods, you can reduce redundancy, ensure data integrity, and create a foundation for meaningful analysis and visualization.

Key Points

Challenge: Daily downloads of the full dataset became impractical as the dataset grew larger.
Solution: Store historical data in a CSV file and append only new data, reducing download time and storage needs.
Filtering: Retain only relevant columns and rows (e.g., glucose readings) while discarding unnecessary metadata and non-glucose entries.
Formatting: Standardize columns, extract additional fields, and add a Treatment column to enrich the dataset.
Validation: Check for missing dates, remove duplicates, and ensure data consistency.
Output: Save processed data in two formats—complete history for records and a 90-day subset for visualization in Tableau.