Background

The problem

CircuitPython runs on embedded hardware, i.e. on a microcontroller board. However we edit our CircuitPython code on a proper computer, i.e. something running Mac, Linux, or even Windows.

To complicate things, a program may be made up of several files of python code if we are structuring things well. On top of that we will almost always be using some numbers of library modules. Additionally we might have sound, image, font, etc. files.

All these files need to be copied to the board's CIRCUITPY drive. Once they are there and you are just editing code it's not a big deal. But what it you want to put the program on a new board? Or the filesystem gets corrupted and you need to rebuild it. You will need to gather up all those files and copy them over. What's that? You were editing directly on CIRCUITPY?

Then there's the fact that the library bundle gets rebuild/updated daily. If you want to keep up with the latest features you'll need to refresh your libraries regularly.

TL;DR: Github Repo

The solution

How do we manage all the files that make up a CircuitPython program?

The first step is to create a complete copy of what should go onto the CIRCUITPY drive. Then it's just a matter of copying it all to CIRCUITPY when we have an update. Libraries still present a complication when they need updating. So this is only a partial solution.

Additionally, you might have file that you don't want on CIRCUITPY. That could be documentation, design files, references images, etc.

With all that in mind, I wrote a small script in Python to take care of it all for me.

Usage

Before digging into the code, let's go over its operation.

Configuration

First there's a file deploy_config.py that lives in the same place as the deploy script and contains overall configuration:

• the directory of your CircuitPython distribution
• the version of CircuitPython to use
• the mountpoint of CIRCUITPY

for example, here's my current config:

config = {
    "dir": "/home/dastels/Documents/CircuitPython",  #
    "version": "6.0",
    "circuitpy": "/media/dastels/CIRCUITPY",
}

So wherever dir is, it will have subdirectories (at least one) corresponding to possible values of version. I've chosen "5.0", "6.0", etc. for these. In those subdirectories is where I keep the various CircuitPython UF2 files for various boards and various versions (extracted) of the bundles.

Each of these can be overridden with command line arguments to the deploy script.

Usage of the script is:

usage: deploy [-h] [-d DIR] [-v VERSION] [-c CIRCUITPY] [-f FILE] [-s] [-l]

Deploy/update a CircuitPython project.

optional arguments:
  -h, --help            show this help message and exit
  -d DIR, --dir DIR     location of CircuitPython files
  -v VERSION, --version VERSION
                        version of CircuitPython to use
  -c CIRCUITPY, --circuitpy CIRCUITPY
                        location of the CIRCUITPY drive
  -f FILE, --file FILE  the single file to copy to CIRCUITPY/code.py
  -s, --subdirs         copy top level subdirectories (e.g. sounds or fonts)
  -l, --updatelibs      update library modules (requires a project manifext.json)

The FILE argument is useful in simple cases when you have a CircuitPython file that isn't called code.py or main.py. For example, the classic LED blink test script: blink.py. Use the command

deploy -f blink.py

to copy blink.py to CIRCUITPY/code.py. You likely won't use this much, and certainly not for your own projects where you'd likely start by writing code.py or main.py.

Typically you'll just use the command:

deploy

which will recursively copy the files in the current directory to CIRCUITPY. If you want to copy subdirectories (e.g. sounds, fonts, images, etc.) use the -s option. If you want to update the contents of the lib folder, use the -l option. See below.

If you place a file named manifest.json in your project directory you can have deploy manage the contents of CIRCUITPY/lib for you, making sure all required modules are present, and updating to the latest verson of the bundle as appropriate. It also let's you skip copying specific project files.

The manifest file is a JSON file containing two arrays: exclude and libs, e.g:

{
    "exclude":[
        "3839d364531413.5ad64cb6a6e75.jpg",
        "test.*",
        ".pytest_cache",
        ".*_example.py",
        "recording_state.py"
    ],
    "libs":[
        "adafruit_adt7410.mpy",
        "adafruit_bitmap_font",
        "adafruit_bno055.mpy",
        "adafruit_bus_device",
        "adafruit_debouncer.mpy",
        "adafruit_display_shapes",
        "adafruit_display_text",
        "adafruit_esp32spi",
        "adafruit_imageload",
        "adafruit_io",
        "adafruit_logging.mpy",
        "adafruit_mcp230xx",
        "adafruit_pyportal.mpy",
        "adafruit_register",
        "adafruit_requests.mpy",
        "adafruit_sdcard.mpy",
        "adafruit_touchscreen.mpy",
        "neopixel.mpy"
    ]
}

The exclude array contains regular expressions (as per the re module) which define files in the project directory that should not be copied to CIRCUITPY. I.e. any files that match any of the exclude regexes won't be copied. In the example, there's a reference image, test files (for pytest) and supporting classes, example files, etc. Think of this like a gitignore file. The manifest.json file is automatically excluded.

The libs array contains the names of files and directories that should be copied into CIRCUITPY/lib. Where do those come from? An extracted bundle that lives in <dir>/<version> (dir and version come from the config file or command line arguments. The script will grab libraries from the most recent extracted bundle directory (in case you didn't clean up old versions for some reason).

So you make note of what files/directories not to copy, and what libraries you need, and then just

deploy -l

If you don't provide a manifest, no libaries will ever be copied, and all project files will be.

Typically you will set up your initial manifest file and run deploy -l to put the library modules in place. After that you will usually just use the -l option when you add to the list of libraries in the manifest file or download & extract a new version of the bundle. You'll also probably seldom use the -s option to avoid taking the time to copy data files in subdirectories, using it only when they get updated. This is because those data files are usually relatively static (and large) compared to code. So most often while working on a project you'll simply use

deploy

to update the desired top level project files, typically the code you're working on.

Without further discussion, here's the code.

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147

#!/usr/bin/python3
# -*-Python-*-

# The MIT License (MIT)
#
# Copyright (c) 2019 Dave Astels
#
# Permission is hereby granted, free of charge, to any person obtaining a copy
# of this software and associated documentation files (the "Software"), to deal
# in the Software without restriction, including without limitation the rights
# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the Software is
# furnished to do so, subject to the following conditions:
#
# The above copyright notice and this permission notice shall be included in
# all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
# THE SOFTWARE.

import os
import argparse
import glob
import json
import re
from deploy_config import config

# Process the command line
parser = argparse.ArgumentParser(description="Deploy/update a CircuitPython project.")
parser.add_argument(
    "-d", "--dir", default=config["dir"], help="location of CircuitPython files"
)
parser.add_argument(
    "-v", "--version", default=config["version"], help="version of CircuitPython to use"
)
parser.add_argument(
    "-c",
    "--circuitpy",
    default=config["circuitpy"],
    help="location of the CIRCUITPY drive",
)
parser.add_argument(
    "-f", "--file", default="", help="the single file to copy to CIRCUITPY/code.py"
)
parser.add_argument(
    "-s",
    "--subdirs",
    action="store_true",
    help="copy top level subdirectories (e.g. sounds or fonts)",
)
parser.add_argument(
    "-l",
    "--updatelibs",
    action="store_true",
    help="update library modules (requires a project manifext.json)",
)

args = parser.parse_args()

manifest = None

# Find the latest bundle
bundles = glob.glob(os.path.join(args.dir, args.version, "*-bundle-*"))
bundles.sort()
bundle_name = bundles[-1]

# Create lib paths
lib_dir = os.path.join(bundle_name, "lib")
cp_lib_dir = os.path.join(args.circuitpy, "lib")


def copy_lib(lib_name):
    print("  " + lib_name)
    os.system("cp -R {0} {1}".format(os.path.join(lib_dir, lib_name), cp_lib_dir))


def included(fname):
    if fname == "manifest.json":
        return False
    if manifest is None:
        return True
    if manifest.get("exclude") is None:
        return True
    return True not in list(
        [re.fullmatch(ex, fname) is not None for ex in manifest["exclude"]]
    )


def cleanup_obsolete_code_file(fname):
    code_path = os.path.join(args.circuitpy, "code.py")
    if fname == "main.py" and os.path.exists(code_path):
        print("main.py detected, removing code.py")
        os.system("rm {0}".format(code_path))
    main_path = os.path.join(args.circuitpy, "main.py")
    if fname == "code.py" and os.path.exists(main_path):
        print("code.py detected, removing main.py")
        os.system("rm {0}".format(main_path))


def replace_file(fname):
    if included(fname):
        dest_path = os.path.join(args.circuitpy, fname)
        cleanup_obsolete_code_file(fname)
        if not os.path.isdir(fname) or args.subdirs:
            print("  " + fname)
            if os.path.exists(dest_path):
                os.system("rm -r {0}".format(dest_path))
            os.system("cp -R {0} {1}".format(fname, args.circuitpy))
    else:
        print("--" + fname)


# Update libraries
try:
    with open("manifest.json") as f:
        manifest = json.loads(f.read())
except IOError:
    print("No manifest.json, skipping library update")

if manifest and args.updatelibs:
    print("Removing old libraries")
    if not os.path.exists(cp_lib_dir):
        print("Creating " + cp_lib_dir)
        os.mkdir(cp_lib_dir)
    os.system("rm -rf {0}".format(os.path.join(cp_lib_dir, "*")))
    print("Copying new libraries from {0}".format(lib_dir))
    for lib in manifest["libs"]:
        copy_lib(lib)

# Copy project files
if args.file == "":
    print("replacing top level files/directories")
    for f in glob.glob("*"):
        replace_file(f)
else:
    code_file = args.file
    if not code_file.endswith(".py"):
        code_file += ".py"
    print("Copy {0} to code.py".format(code_file))
    os.system("cp {0} {1}".format(code_file, os.path.join(args.circuitpy, "code.py")))

os.system("sync")