Interface data storage in PROGMEM (#71)

Adds a webpack plugin to package interface as PROGMEM into a header file in the framework.
Adds a build flag to optionally enable serving from PROGMEM or SPIFFS as required
Adds documentation changes to describe changes
This commit is contained in:
rjwats 2019-12-29 17:54:12 +00:00 committed by GitHub
parent 14f50c1e31
commit bcfeef8004
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
20 changed files with 324 additions and 85 deletions

3
.gitignore vendored
View File

@ -1,10 +1,9 @@
.pio
.pioenvs
.piolibdeps
.clang_complete
.gcc-flags.json
*Thumbs.db
/data/www
/lib/framework/WWWData.h
/interface/build
/interface/node_modules
.vscode

View File

@ -1,6 +1,10 @@
language: python
python:
- "2.7"
- "3.8"
before_install:
- nvm install 10.15.3
- nvm use 10.15.3
sudo: false
cache:

View File

@ -30,7 +30,6 @@ You will need the following before you can get started.
* [PlatformIO](https://platformio.org/) - IDE for development
* [Node.js](https://nodejs.org) - For building the interface with npm
* Bash shell, or [Git Bash](https://gitforwindows.org/) if you are under windows
### Building and uploading the firmware
@ -74,35 +73,15 @@ Alternatively run the 'upload' target:
platformio run -t upload
```
### Building the interface
### Building & uploading the interface
The interface has been configured with create-react-app and react-app-rewired so the build can customized for the target device. The large artefacts are gzipped and source maps and service worker are excluded from the production build. This reduces the production build to around ~200k, which easily fits on the device.
Change to the ['interface'](interface) directory with your bash shell (or Git Bash) and use the standard commands you would with any react app built with create-react-app:
#### Change to interface directory
```bash
cd interface
```
#### Download and install the node modules
```bash
npm install
```
#### Build the interface
```bash
npm run build
```
> **Note**: The build command will also delete the previously built interface, in the ['data/www'](data/www) directory, replacing it with the freshly built one ready to upload to the device.
The interface will be automatically built by PlatformIO before it builds the firmware. The project can be configured to serve the interface from either SPIFFS or PROGMEM as your project requires. The default configuration is to serve the content from SPIFFS which requires an additional upload step which is documented below.
#### Uploading the file system image
The compiled user interface may be uploaded to the device by pressing the "Upload File System image" button:
If service content from SPIFFS (default), build the project first. Then the compiled interface may be uploaded to the device by pressing the "Upload File System image" button:
![uploadfs](/media/uploadfs.png?raw=true "uploadfs")
@ -112,15 +91,43 @@ Alternatively run the 'uploadfs' target:
platformio run -t uploadfs
```
#### Serving the interface from PROGMEM
You can configure the project to serve the interface from PROGMEM by uncommenting the -D PROGMEM_WWW build flag in ['platformio.ini'](platformio.ini) then re-building and uploading the firmware to the device.
Be aware that this will consume ~150k of program space which can be especially problematic if you already have a large build artefact or if you have added large javascript dependencies to the interface. The ESP32 binaries are large already, so this will be a problem if you are using one of these devices and require this type of setup.
A method for working around this issue can be to reduce the amount of space allocated to SPIFFS by configuring the device to use a differnt strategy partitioning. If you don't require SPIFFS other than for storing config one approach might be to configure a minimal SPIFFS partition.
For a ESP32 (4mb variant) there is a handy "min_spiffs.csv" partition table which can be enabled easily:
```yaml
[env:node32s]
board_build.partitions = min_spiffs.csv
platform = espressif32
board = node32s
```
This is largley left as an exersise for the reader as everyone's requirements will vary.
### Running the interface locally
You can run a local development server to allow you preview changes to the front end without the need to upload a file system image to the device after each change. Change to the interface directory and run the following command:
You can run a local development server to allow you preview changes to the front end without the need to upload a file system image to the device after each change.
Change to the ['interface'](interface) directory with your bash shell (or Git Bash) and use the standard commands you would with any react app built with create-react-app:
```bash
cd interface
```
Install the npm dependencies, if required and start the development server:
```bash
npm install
npm start
```
> **Note**: To run the interface locally you will need to modify the endpoint root path and enable CORS.
> **Note**: To run the interface locally you may need to modify the endpoint root path and enable CORS.
#### Changing the endpoint root
@ -141,7 +148,9 @@ You can enable CORS on the back end by uncommenting the -D ENABLE_CORS build fla
## Device Configuration
As well as containing the interface, the SPIFFS image (in the ['data'](data) folder) contains a JSON settings file for each of the configurable features. The config files can be found in the ['data/config'](data/config) directory:
The SPIFFS image (in the ['data'](data) folder) contains a JSON settings file for each of the configurable features.
The config files can be found in the ['data/config'](data/config) directory:
File | Description
---- | -----------
@ -173,30 +182,31 @@ It is recommended that you change the JWT secret and user credentials from their
This project supports ESP8266 and ESP32 platforms. To support OTA programming, enough free space to upload the new sketch and file system image will be required. It is recommended that a board with at least 2mb of flash is used.
By default, the target device is "esp12e". This is a common ESP8266 variant with 4mb of flash:
The pre-configured environments are "esp12e" and "node32s". These are common ESP8266/ESP32 variants with 4mb of flash:
![ESP12E](/media/esp12e.jpg?raw=true "ESP12E")
![ESP12E](/media/esp12e.jpg?raw=true "ESP12E") ![ESP32](/media/esp32.jpg?raw=true "ESP32")
The settings file ['platformio.ini'](platformio.ini) configures the platform and board:
The settings file ['platformio.ini'](platformio.ini) configures the supported environments. Modify these, or add new environments for the devides you need to support. The default environments are as follows:
```
```yaml
[env:esp12e]
platform = espressif8266
board = esp12e
```
board_build.f_cpu = 160000000L
If you want to build for an ESP32 device, all you need to do is re-configure ['platformio.ini'](platformio.ini) with your devices settings.
![ESP32](/media/esp32.jpg?raw=true "ESP32")
Building for the common esp32 "node32s" board for example requires the following configuration:
```
[env:node32s]
platform = espressif32
board = node32s
```
If you want to build for a different device, all you need to do is re-configure ['platformio.ini'](platformio.ini) and select an alternative environment by modifying the default_envs variable. Building for the common esp32 "node32s" board for example:
```yaml
[platformio]
;default_envs = esp12e
default_envs = node32s
```
## Customizing and theming
The framework, and MaterialUI allows for a reasonable degree of customization with little effort.
@ -274,7 +284,11 @@ void setup() {
Serial.begin(SERIAL_BAUD_RATE);
// start the file system (must be done before starting the framework)
#ifdef ESP32
SPIFFS.begin(true);
#elif defined(ESP8266)
SPIFFS.begin();
#endif
// start the framework and demo project
esp8266React.begin();

View File

@ -1,7 +1,8 @@
const ManifestPlugin = require('webpack-manifest-plugin');
const WorkboxWebpackPlugin = require('workbox-webpack-plugin');
const MiniCssExtractPlugin = require('mini-css-extract-plugin');
const CompressionPlugin = require("compression-webpack-plugin");
const CompressionPlugin = require('compression-webpack-plugin');
const ProgmemGenerator = require('./progmem-generator.js');
const path = require('path');
const fs = require('fs');
@ -21,6 +22,9 @@ module.exports = function override(config, env) {
miniCssExtractPlugin.options.filename = "css/[id].[contenthash:4].css";
miniCssExtractPlugin.options.chunkFilename = "css/[id].[contenthash:4].c.css";
// build progmem data files
config.plugins.push(new ProgmemGenerator({ outputPath: "../lib/framework/WWWData.h", bytesPerLine: 20 }));
// add compression plugin, compress javascript
config.plugins.push(new CompressionPlugin({
filename: "[path].gz[query]",

View File

@ -8598,11 +8598,18 @@
"integrity": "sha512-jYdeOMPy9vnxEqFRRo6ZvTZ8d9oPb+k18PKoYNYUe2stVEBPPwsln/qWzdbmaIvnhZ9v2P+CuecK+fpUfsV2mA=="
},
"mime-types": {
"version": "2.1.24",
"resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.24.tgz",
"integrity": "sha512-WaFHS3MCl5fapm3oLxU4eYDw77IQM2ACcxQ9RIxfaC3ooc6PFuBMGZZsYpvoXS5D5QTWPieo1jjLdAm3TBP3cQ==",
"version": "2.1.25",
"resolved": "https://registry.npmjs.org/mime-types/-/mime-types-2.1.25.tgz",
"integrity": "sha512-5KhStqB5xpTAeGqKBAMgwaYMnQik7teQN4IAzC7npDv6kzeU6prfkR67bc87J1kWMPGkoaZSq1npmexMgkmEVg==",
"requires": {
"mime-db": "1.40.0"
"mime-db": "1.42.0"
},
"dependencies": {
"mime-db": {
"version": "1.42.0",
"resolved": "https://registry.npmjs.org/mime-db/-/mime-db-1.42.0.tgz",
"integrity": "sha512-UbfJCR4UAVRNgMpfImz05smAXK7+c+ZntjaA26ANtkXLlOe947Aag5zdIcKQULAiF9Cq4WxBi9jUs5zkA84bYQ=="
}
}
},
"mimic-fn": {
@ -13483,6 +13490,11 @@
"camelcase": "^5.0.0",
"decamelize": "^1.2.0"
}
},
"zlib": {
"version": "1.0.5",
"resolved": "https://registry.npmjs.org/zlib/-/zlib-1.0.5.tgz",
"integrity": "sha1-bnyXL8NxxkWmr7A6sUdp3vEU/MA="
}
}
}

View File

@ -7,6 +7,7 @@
"@material-ui/icons": "^4.5.1",
"compression-webpack-plugin": "^2.0.0",
"jwt-decode": "^2.2.0",
"mime-types": "^2.1.25",
"moment": "^2.24.0",
"notistack": "^0.9.6",
"prop-types": "^15.7.2",
@ -17,11 +18,12 @@
"react-material-ui-form-validator": "^2.0.9",
"react-router": "^5.1.1",
"react-router-dom": "^5.1.1",
"react-scripts": "3.0.1"
"react-scripts": "3.0.1",
"zlib": "^1.0.5"
},
"scripts": {
"start": "react-app-rewired start",
"build": "react-app-rewired build && rm -rf ../data/www && cp -r build ../data/www",
"build": "react-app-rewired build",
"test": "react-app-rewired test --env=jsdom",
"eject": "react-scripts eject"
},

View File

@ -0,0 +1,122 @@
const { resolve, relative, sep } = require('path');
const { readdirSync, existsSync, unlinkSync, readFileSync, createWriteStream } = require('fs');
var zlib = require('zlib');
var mime = require('mime-types');
const ARDUINO_INCLUDES = "#include <Arduino.h>\n\n";
function getFilesSync(dir, files = []) {
readdirSync(dir, { withFileTypes: true }).forEach(entry => {
const entryPath = resolve(dir, entry.name);
if (entry.isDirectory()) {
getFilesSync(entryPath, files);
} else {
files.push(entryPath);
}
})
return files;
}
function coherseToBuffer(input) {
return Buffer.isBuffer(input) ? input : Buffer.from(input);
}
function cleanAndOpen(path) {
if (existsSync(path)) {
unlinkSync(path);
}
return createWriteStream(path, { flags: "w+" });
}
class ProgmemGenerator {
constructor(options = {}) {
const { outputPath, bytesPerLine = 20, indent = " ", includes = ARDUINO_INCLUDES } = options;
this.options = { outputPath, bytesPerLine, indent, includes };
}
apply(compiler) {
compiler.hooks.emit.tapAsync(
{ name: 'ProgmemGenerator' },
(compilation, callback) => {
const { outputPath, bytesPerLine, indent, includes } = this.options;
const fileInfo = [];
const writeStream = cleanAndOpen(resolve(compilation.options.context, outputPath));
try {
const writeIncludes = () => {
writeStream.write(includes);
}
const writeFile = (relativeFilePath, buffer) => {
const variable = "ESP_REACT_DATA_" + fileInfo.length;
const mimeType = mime.lookup(relativeFilePath);
var size = 0;
writeStream.write("const uint8_t " + variable + "[] PROGMEM = {");
const zipBuffer = zlib.gzipSync(buffer);
zipBuffer.forEach((b) => {
if (!(size % bytesPerLine)) {
writeStream.write("\n");
writeStream.write(indent);
}
writeStream.write("0x" + ("00" + b.toString(16).toUpperCase()).substr(-2) + ",");
size++;
});
if (size % bytesPerLine) {
writeStream.write("\n");
}
writeStream.write("};\n\n");
fileInfo.push({
uri: '/' + relativeFilePath.replace(sep, '/'),
mimeType,
variable,
size
});
};
const writeFiles = () => {
// process static files
const buildPath = compilation.options.output.path;
for (const filePath of getFilesSync(buildPath)) {
const readStream = readFileSync(filePath);
const relativeFilePath = relative(buildPath, filePath);
writeFile(relativeFilePath, readStream);
}
// process assets
const { assets } = compilation;
Object.keys(assets).forEach((relativeFilePath) => {
writeFile(relativeFilePath, coherseToBuffer(assets[relativeFilePath].source()));
});
}
const generateWWWClass = () => {
return `typedef std::function<void(const String& uri, const String& contentType, const uint8_t * content, size_t len)> RouteRegistrationHandler;
class WWWData {
${indent}public:
${indent.repeat(2)}static void registerRoutes(RouteRegistrationHandler handler) {
${fileInfo.map(file => `${indent.repeat(3)}handler("${file.uri}", "${file.mimeType}", ${file.variable}, ${file.size});`).join('\n')}
${indent.repeat(2)}}
};
`;
}
const writeWWWClass = () => {
writeStream.write(generateWWWClass());
}
writeIncludes();
writeFiles();
writeWWWClass();
writeStream.on('finish', () => {
callback();
});
} finally {
writeStream.end();
}
}
);
}
}
module.exports = ProgmemGenerator;

View File

@ -3,20 +3,20 @@
font-family: 'Roboto';
font-style: normal;
font-weight: 300;
src: local('Roboto Light'), local('Roboto-Light'), url(../fonts/ro-li.w2) format('woff2');
src: local('Roboto Light'), local('Roboto-Light'), url(../fonts/li.woff2) format('woff2');
unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2212, U+2215;
}
@font-face {
font-family: 'Roboto';
font-style: normal;
font-weight: 400;
src: local('Roboto'), local('Roboto-Regular'), url(../fonts/ro-re.w2) format('woff2');
src: local('Roboto'), local('Roboto-Regular'), url(../fonts/re.woff2) format('woff2');
unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2212, U+2215;
}
@font-face {
font-family: 'Roboto';
font-style: normal;
font-weight: 500;
src: local('Roboto Medium'), local('Roboto-Medium'), url(../fonts/ro-me.w2) format('woff2');
src: local('Roboto Medium'), local('Roboto-Medium'), url(../fonts/me.woff2) format('woff2');
unicode-range: U+0000-00FF, U+0131, U+0152-0153, U+02BB-02BC, U+02C6, U+02DA, U+02DC, U+2000-206F, U+2074, U+20AC, U+2122, U+2212, U+2215;
}

View File

@ -13,8 +13,8 @@
#define DNS_PORT 53
#define AP_DEFAULT_SSID "ssid"
#define AP_DEFAULT_PASSWORD "password"
#define AP_DEFAULT_SSID "ESP8266-React"
#define AP_DEFAULT_PASSWORD "esp-react"
#define AP_SETTINGS_FILE "/config/apSettings.json"
#define AP_SETTINGS_SERVICE_PATH "/rest/apSettings"

View File

@ -13,13 +13,37 @@ ESP8266React::ESP8266React(AsyncWebServer* server, FS* fs) :
_ntpStatus(server, &_securitySettingsService),
_apStatus(server, &_securitySettingsService),
_systemStatus(server, &_securitySettingsService) {
#ifdef PROGMEM_WWW
// Serve static resources from PROGMEM
WWWData::registerRoutes(
[server, this](const String& uri, const String& contentType, const uint8_t* content, size_t len) {
ArRequestHandlerFunction requestHandler = [contentType, content, len](AsyncWebServerRequest* request) {
AsyncWebServerResponse* response = request->beginResponse_P(200, contentType, content, len);
response->addHeader("Content-Encoding", "gzip");
request->send(response);
};
server->on(uri.c_str(), HTTP_GET, requestHandler);
// Serving non matching get requests with "/index.html"
// OPTIONS get a straight up 200 response
if (uri.equals("/index.html")) {
server->onNotFound([requestHandler](AsyncWebServerRequest* request) {
if (request->method() == HTTP_GET) {
requestHandler(request);
} else if (request->method() == HTTP_OPTIONS) {
request->send(200);
} else {
request->send(404);
}
});
}
});
#else
// Serve static resources from /www/
server->serveStatic("/js/", SPIFFS, "/www/js/");
server->serveStatic("/css/", SPIFFS, "/www/css/");
server->serveStatic("/fonts/", SPIFFS, "/www/fonts/");
server->serveStatic("/app/", SPIFFS, "/www/app/");
server->serveStatic("/favicon.ico", SPIFFS, "/www/favicon.ico");
// Serving all other get requests with "/www/index.htm"
// OPTIONS get a straight up 200 response
server->onNotFound([](AsyncWebServerRequest* request) {
@ -31,6 +55,7 @@ ESP8266React::ESP8266React(AsyncWebServer* server, FS* fs) :
request->send(404);
}
});
#endif
// Disable CORS if required
#if defined(ENABLE_CORS)

View File

@ -4,9 +4,9 @@
#include <Arduino.h>
#ifdef ESP32
#include <WiFi.h>
#include <AsyncTCP.h>
#include <SPIFFS.h>
#include <WiFi.h>
#elif defined(ESP8266)
#include <ESP8266WiFi.h>
#include <ESPAsyncTCP.h>
@ -26,6 +26,10 @@
#include <WiFiSettingsService.h>
#include <WiFiStatus.h>
#ifdef PROGMEM_WWW
#include <WWWData.h>
#endif
class ESP8266React {
public:
ESP8266React(AsyncWebServer* server, FS* fs);
@ -52,6 +56,7 @@ class ESP8266React {
NTPStatus _ntpStatus;
APStatus _apStatus;
SystemStatus _systemStatus;
};
#endif

View File

@ -17,6 +17,9 @@ void SecuritySettingsService::readFromJsonObject(JsonObject& root) {
for (JsonVariant user : root["users"].as<JsonArray>()) {
_users.push_back(User(user["username"], user["password"], user["admin"]));
}
} else {
_users.push_back(User(DEFAULT_ADMIN_USERNAME, DEFAULT_ADMIN_USERNAME, true));
_users.push_back(User(DEFAULT_GUEST_USERNAME, DEFAULT_GUEST_USERNAME, false));
}
}

View File

@ -4,6 +4,9 @@
#include <AdminSettingsService.h>
#include <SecurityManager.h>
#define DEFAULT_ADMIN_USERNAME "admin"
#define DEFAULT_GUEST_USERNAME "guest"
#define SECURITY_SETTINGS_FILE "/config/securitySettings.json"
#define SECURITY_SETTINGS_PATH "/rest/securitySettings"

View File

@ -1,35 +1,40 @@
; PlatformIO Project Configuration File
;
; Build options: build flags, source filter
; Upload options: custom upload port, speed and extra flags
; Library options: dependencies, extra library storages
; Advanced options: extra scripting
;
; Please visit documentation for the other options and examples
; http://docs.platformio.org/page/projectconf.html
[env:esp12e]
platform = espressif8266
board = esp12e
board_build.f_cpu = 160000000L
extra_scripts = pre:timelib_fix.py
framework = arduino
monitor_speed = 115200
; Uncomment & modify the lines below in order to configure OTA updates
;upload_flags =
; --port=8266
; --auth=esp-react
;upload_port = 192.168.0.11
[platformio]
default_envs = esp12e
;default_envs = node32s
[env]
build_flags=
-D NO_GLOBAL_ARDUINOOTA
; Uncomment ENABLE_CORS to enable Cross-Origin Resource Sharing (required for local React development)
;-D ENABLE_CORS
-D CORS_ORIGIN=\"http://localhost:3000\"
; Uncomment PROGMEM_WWW to enable the storage of the WWW data in PROGMEM
;-D PROGMEM_WWW
; Uncomment & modify the lines below in order to configure OTA updates
;upload_flags =
; --port=8266
; --auth=esp-react
;upload_port = 192.168.0.11
framework = arduino
monitor_speed = 115200
extra_scripts =
pre:scripts/timelib_fix.py
pre:scripts/build_interface.py
lib_deps =
NtpClientLib@>=2.5.1,<3.0.0
ArduinoJson@>=6.0.0,<7.0.0
ESP Async WebServer@>=1.2.0,<2.0.0
AsyncTCP@>=1.0.3,<2.0.0
[env:esp12e]
platform = espressif8266
board = esp12e
board_build.f_cpu = 160000000L
[env:node32s]
;board_build.partitions = min_spiffs.csv
platform = espressif32
board = node32s

View File

@ -0,0 +1,36 @@
from pathlib import Path
from shutil import copytree
from shutil import rmtree
from subprocess import check_output, Popen, PIPE, STDOUT, CalledProcessError
from os import chdir
Import("env")
def flagExists(flag):
buildFlags = env.ParseFlags(env["BUILD_FLAGS"])
for define in buildFlags.get("CPPDEFINES"):
if (define == flag or (isinstance(define, list) and define[0] == flag)):
return True
def buildWeb():
chdir("interface")
print("Building interface with npm")
try:
env.Execute("npm install")
env.Execute("npm run build")
buildPath = Path("build")
wwwPath = Path("../data/www")
if wwwPath.exists() and wwwPath.is_dir():
rmtree(wwwPath)
if not flagExists("PROGMEM_WWW"):
print("Copying interface to data directory")
copytree(buildPath, wwwPath)
finally:
chdir("..")
if (len(BUILD_TARGETS) == 0 or "upload" in BUILD_TARGETS):
buildWeb()
else:
print("Skipping build interface step for target(s): " + ", ".join(BUILD_TARGETS))

View File

@ -1,6 +1,7 @@
import os
import sys
import re
Import("env")
# Find files under 'root' of a given 'fileName' in directories matching 'subDirectoryPattern'
@ -26,14 +27,14 @@ def deleteTimeHeader(libDepsDir):
if numDeletionCandidates == 1:
os.remove(deletionCandidates[0])
elif numDeletionCandidates > 1:
os.write(2, 'Can\'t delete Time.h, more than one instance found:\n' + '\n'.join(deletionCandidates))
os.write(2, "Can\'t delete Time.h, more than one instance found:\n" + "\n".join(deletionCandidates))
sys.exit(1)
# old lib deps directory
deleteTimeHeader(os.path.join(env.subst('$PROJECT_DIR'), '.piolibdeps'))
deleteTimeHeader(os.path.join(env.subst("$PROJECT_DIR"), ".piolibdeps"))
# pre 4.x lib deps directory
deleteTimeHeader(os.path.join(env.subst('$PROJECTLIBDEPS_DIR'), env.subst('$PIOENV')))
deleteTimeHeader(os.path.join(env.subst("$PROJECTLIBDEPS_DIR"), env.subst("$PIOENV")))
# >4.x lib deps directory
deleteTimeHeader(os.path.join(env.subst('$PROJECT_LIBDEPS_DIR'), env.subst('$PIOENV')))
deleteTimeHeader(os.path.join(env.subst("$PROJECT_LIBDEPS_DIR"), env.subst("$PIOENV")))

View File

@ -13,7 +13,11 @@ void setup() {
Serial.begin(SERIAL_BAUD_RATE);
// start the file system (must be done before starting the framework)
#ifdef ESP32
SPIFFS.begin(true);
#elif defined(ESP8266)
SPIFFS.begin();
#endif
// start the framework and demo project
esp8266React.begin();