In a recent project, the Boston Engineering team implemented an embedded device that aggregates sensor data and stores it until it can be uploaded to a central server. Due to power and cost constraints, this system is using an embedded microcontroller. Compared to more capable Linux systems, microcontroller based embedded systems applications often have resource constraints to work within. These resource constraints can be challenging, but with proper preparation and testing, data can be safely and logically stored for later retrieval by another process, even after power loss or system power-off.
This document will discuss how we used the SPIFFS filesystem to enable storage of sensor data using a SPI Flash storage, with discussion of some challenges that were found during the process.
The Problem: log entries need to be committed at a high frequency, but the performance of SPIFFS may decrease enough that it will be unable to commit incoming data in a timely manner, which may result in data loss.
The Solution: The several approaches Boston Engineering has compiled in this technical white paper!
Boston Engineering is a leader in the development of custom embedded hardware and software systems. Our integrated, cross functional team of electrical, software, and mechanical engineers will advise, direct, and manage any computing development project.
In this impactful whitepaper, explore the steps used to increase Performance for Frequent writes:
Download for groundbreaking insights!
Approaches
A few approaches to storing data in the device’s flash memory were considered:
Raw Data Storage in Flash In this approach, the data is written directly to certain memory addresses on the flash chip. The application used to communicate with the flash storage must keep track of the memory addresses that are in use and be certain that the address spaces of separate data do not overlap. The application is also responsible for erasing regions in memory before reuse.
This approach works well for small amounts of data such as device serial numbers or configuration data but is cumbersome to manage if the application demands larger amounts of data or multiple data types that must occupy their own memory space without overwriting others.
Use of a File System Library
Another approach is to use a filesystem library. Filesystem software will manage files as requested by the application. These files can be in a single memory address space or can be fragmented across several address spaces. This allows the application developer to create and manage multiple data types by referencing each type by its file name, which greatly simplifies data management tasks for the developer.
In the end we chose to utilize a filesystem, in this case SPIFFS, for data storage in the device.
Why We Chose SPIFFS
Through testing and optimization, we were able to exceed our target performance with SPIFFS, an open-source filesystem targeted towards embedded systems containing a SPI flash in the hardware design. SPIFFS offers the developer several notable advantages:
Open Source
SPIFFS is a commonly used and readily available open-source filesystem library with a generous license (MIT). Open-source software can have an active development community leading to strong code and functionality. Using a library that has been evaluated and used by many developers over time can reduce effort and add confidence rather than writing and maintaining the library in-house.
Why We Chose SPIFFS
Through testing and optimization, we were able to exceed our target performance with SPIFFS, an open-source filesystem targeted towards embedded systems containing a SPI flash in the hardware design. SPIFFS offers the developer several notable advantages:
Open Source
SPIFFS is a commonly used and readily available open-source filesystem library with a generous license (MIT). Open-source software can have an active development community leading to strong code and functionality. Using a library that has been evaluated and used by many developers over time can reduce effort and add confidence rather than writing and maintaining the library in-house.
Targeted for SPI flash
SPI flash chips have several characteristics that must be considered for use. In general:
SPIFFS manages all these concerns internally, giving the developer an API to focus on the application rather than the internals of managing the SPI flash.
POSIX compliant
The Portable Operating System Interface (POSIX) is a standard for maintaining compatibility between operating systems. Complying with the POSIX standard allows for developers to port their applications to other systems easily. In addition, a general knowledge of the commands available in the POSIX standard simplifies the bring-up of libraries such as SPIFFS. Once the library has been installed and configured, commonly used commands are available for the developer.
Problem Statement
When using the SPIFFS open-source library for frequent data logging operations, performance appears to degrade as more data is written to the filesystem. In the case where log entries need to be committed at a high frequency, the performance of SPIFFS may decrease enough that it will be unable to commit incoming data in a timely manner, which may result in data loss.
For a copy of the the complete publication, visit the Embedded Systems High Frequency Data Logging using SPIFFs download page.
As the engineering landscape continues to evolve, Boston Engineering is committed to proactively integrating cutting-edge technologies and methodologies into the solutions it delivers, ensuring that clients stay at the forefront of industry trends and maintain a competitive edge.
About Boston Engineering Embedded Systems
Boston Engineering is a leader in the development of custom embedded hardware and software systems. Our integrated, cross functional team of electrical, software, and mechanical engineers will advise, direct, and manage any computing development project. Whether the challenge is to modernize components, increase reliability, improve performance, or synchronize I/O options, Boston Engineering thrives on solving the toughest challenges.
Embedded Systems is one of Boston Engineering’s Centers of Excellence. Contact us to learn more about how our Centers of Excellence define and support the commitment we make to our clients and the organizations we serve.
Ready to learn more about Boston Engineering?
For almost three decades, Boston Engineering has designed, developed, and optimized devices and technologies the medical, Commercial, and Defense communities rely on to improve the way people work and live. We provide solutions to the challenges society faces.
Our expertise includes industrial design and product redesign, sensors and control systems, robotics technical innovation, and digital software solutions.
Imagine your Impact: Stay up-to-date with the latest insights and trends we're watching. Add your email address below and sign up for a monthly summary of our most impactful posts!