As we Arduino environment for 8051 pulled, or OS on a single process

In the summer of 2016, we released available for sale to our new Board for the development of Z-Wave devices Z-Uno. This is an absolutely innovative device, unique in the world Z-Wave yet. Given the large number of programmers ' chips, I decided to share some of the solutions used in the Z-Uno.

In short, we have simplified the cooperative OS 1 process at a microcontroller of the 8051 family with an API similar to the Arduino.

Let's just answer the main question: WHY?

As I already wrote are Z-Wave devices is very difficult. Requires not only skills of work with microcontrollers, but also special software and programmers.

Our goal was to give the user a relatively simple means for creating Z-Wave devices without using expensive and specific tools and pieces of iron. In addition, the Protocol Z-Wave is not too obvious, and we wanted to "hide under the hood" all the details of the Protocol, leaving to the user only the basic essence.

As we would like to give the user the maximum number the hardware capabilities of the chip Z-Wave (legs, hardware drivers, tires, ...), it was decided to base the Arduino environment. She is popular, just gives the opportunity to work with the hardware of the microcontroller and uses a slightly simplified C++ (not all the chips of the pros are available there). And not only the API style (list of common Arduino functions for the treatment to the prostate), but also the IDE. But we have a caveat — a lot of work to do for the user, especially radio exchange, i.e., periodically need to take control and do all the "dirty work", returning control when we are not busy with radio service and processing teams.

In addition, we may not distribute the library Z-Wave as is (the claim of the owner of the Protocol associated with the NDA), and even though the network is full firmware in .bin or .hex format (for OTA updates to devices, for example), to include the library in your Arduino environment, we could not. Considering all the above, we just need to isolate user code from the code packet Z-Wave.

So we did OS 1 process, giving the user a simple API to the developer in the style of Arduino.

About using Z-Uno (and older versions), I wrote a separate article. Also on the GT is other articles. Here will be described the implementation details of the internals of the Z-Uno. Dear reader, welcome to our "behind the scenes".

the

Architecture

If very briefly, the Z-Uno consists of 4 parts:

the
1. Loader (bootloader) that allows you to change the firmware. Almost all devices with OTA update have this.
   
   
the
2. Stack Z-Wave from Sigma Designs is a library linked with the next level.
   
   
the
3. Implementation of command classes Z-Wave basic functions, as well as all the work with skycom (fill in the sketch and the response side Arduino-like API). This part we call "loader"sketch.
   
   
the
4. Custom sketch loaded by the user via the Arduino IDE — just like with any of the Arduino boards.

the

Multitasking

The transfer of control from the sketch to the bootloader sketch takes place voluntarily. However, a long stay in the sketch (more than 10 MS) can screw up the exchange of data via radio. In the opposite direction (from the loader sketch in the sketch) the office also dispatched when the loader is idle sketch. Thus, even handing control to the sketch, a occasionally interrupts briefly return control to the loader sketch. So here is a simple cooperative OS.

the

the entry Point in the user process (sketch)

Classic C-interface the program starts with main(). In the Arduino sketches instead uses the setup() and loop(). We decided to adapt this Convention — at the start of the Z-Uno in the initialization process of iron called setup(), then all the time that the stack Z-Wave and loader in the sketch are called loop(). Everything seems easy. There is another point of contact with the user sketch associated with the implementation of the interaction on the Z-Wave network: getter and setter methods. About them is below.
the

Representation in Z-Wave and channels

The legs of the Z-Uno a lot of iron is different, you can connect a lot. But it's not just Arduino, we have Z-Wave for some reason. The main objective of Z-Uno — a map of peripherals connected to the legs of the Z-Uno in Z-Wave entity and Vice versa. Since Z-Wave devices can have many different features, we decided to give the user access to multiple entities. To simplify, we decided to create a channel for the entity (I will not go to the details Z-Wave, there were other ways to do it). Each channel a type, depending on user settings, and implemented within the appropriate Classes of Commands (Command Classes). Such types, we have until four: binary sensor (command class Binary Sensor), sensor multilevel (Multilevel Sensor), relays (Binary Switch), dimmer (Switch Multilevel). In the future there will be a counter (Meter) and locks (Door Lock).

All these classes implement the commands retrieve the current value (Get), relays and dimmers are also implemented setting the value (Set). Receiving commands and sending reports does our code — loader sketch, but these values need to take from the user sketch and give it to him. This interaction we have implemented via getter/setter mechanism. In the description of each channel the user must specify the function to use as getter and setter methods.

the

Getter and Setter

For correct operation in the Z-Wave network, we need to request from the other network devices to respond promptly to the queries of the current States and process commands to set new values. For example, the motion sensor could send Set command to the relay base Z-Uno. Or the controller could ask us about the current status of the relay or current sensor value connected to Z-Uno. All these commands we need to execute quickly, and the value we should get from the user code, there came to pass the new values for channels. "Quickly" is a relative term. We found that it is sufficient to wait for the custom code out of loop() or call delay(). Thus, the getter and setter methods are run only when the user code is not executed.

Now, in order analyze the blocks of the resulting system.

the

Build and load the code in the Z-Uno

Since we decided to use the Arduino IDE, we needed to create a custom package compiler, loader, libraries and include files, which is installed through the Board Manager environment Arduino IDE. Here we describe the installation process for those who are not familiar with it.

the

Compiler

Chip Z-Wave is based on the 8051 architecture, i.e., the standard avr-gcc does not suit us. Anything interesting and at the same time open for 8051 compiler SDCC in addition to sdcc.sourceforge.net we have found. Alas, he realizes pure C, no "pros". But the compilation of the C code he is doing quite well, although not as good as the expensive Keil (which is used to create all Z-Wave devices, including our part of the code Z-Uno). Lucky for us, the creators SDCC advance provided a lot of options which we have used: restriction on use of Code Space, IDATA, XDATA, addresses the interrupt vectors... more On this later in the section resource distribution.

the

C++

Most of the libraries for Arduino are now using C++, but rather some of its syntax. As already mentioned, to compile C++ SDCC does not know how. But many of the Arduino libraries use classes, inheritance and polymorphism. We tried out different options, starting with good old cfront and ending newfangled clang. After long hesitation it was decided to take clang and use it to parse the custom code and then create pure C-code that will gather SDCC. Thus, we use clang as the compiler With++ the code in C, and not as a full compiler. On the same principle worked the first C++ compiler is cfront mentioned earlier.
There arises immediately the question: "Why did you go so archaic and strange way." The answer is very simple: create a full-fledged C++ compiler for 8051 would require a lot of time, I can even say sooo much time disproportionately more than the time we took on the whole project Z-Uno. In addition, we tried to limit the supported semantic designs, various "features" of C++, and that's why I called our translator uCxx (abbreviated as u=[mj:u=micro). Strictly speaking, our translator supports a very limited dialect of C++. uCxx at the moment can't overload operators that don't know nothing about templates, also does not work with links, does not support multiple inheritance, he never heard about the new and delete operators. His whole gentleman's set is limited by polymorphism at the level of classes and virtual functions, but this set is enough for porting most Arduinо'vskih library with an almost complete preservation of their interface. In addition, uCxx makes some "chips" that are there. For example especially for Z-Uno he is able to realign the pins with a dedicated port, therefore, to provide maximum speed control pin, can fill noami (NOP instruction) are necessary parts of the code etc. We left immediately from the universality and tried to make it special and as fast as possible at design-time decision.

There's a lot of technical details about code generation

Now briefly try to describe the principles of operation uCxx. First of all what it is !? We use a specially patched version of libclang (yet there's still a lot of small flaws, such as the definition of a binary/unary operator and similar things that had little to improve library), binding libclang (also had to edit to match the patched library) for Python. The main development language uCxx, so is Python. Python has also been chosen to simplify development and to buy time. Yes, uCxx — it's just a Python script working libclang, but nevertheless, Python-uCxx code are converted to binary build using pyinstaller package and the end user does not need to know anything about Python, its execution environment, and additional libraries.

Will try to show how it works uCxx. First, the user sketch is the analysis phase where you define all used headers and it generates a list of additional kernel files/libraries that must be included in the compilation (the same is true for native Arduino'vsky preprocessor). The file .ino is sent to the preprocessor: use a third-party — sdcpp (part of the SDCC compiler). Thereafter, the resulting. cpp file being pushed into clang, which at the output produces Abstract Syntax Tree (AST) of the entire file. It is at this stage identifies all syntax errors. Looks like the main part of the AST for the source code can be seen in a special debugging files which have the suffix _ast.txt. Received AST tree parsed code uCxx. In fact it is the bypass of a large tree. For each found class, a special structure that stores all the data of a class object. For each method defined by its new name that is generated based on the name of the parent class, the number and type of input parameters. This technique is common for C++compilers and is called "mangling". In uCxx uses a different algorithm for constructing such names, because the built-in library of clang, the algorithm proved to be inefficient for the designers to correct it was much more difficult than writing your own. In every non-static class method is also added — the first parameter is dereferenced in the future as this, which is also the standard approach for OOP-compiler. For example, in languages like Python this syntax is familiar to the user.
At the core of our compiler is an implementation of the virtual methods. In ishh they are implemented using tables of virtual methods, which is formed for each class statically at compile time. The table is populated by function pointers. Function, in this case, we call translated to the C language method of a class. For the names of these functions introduced a special order relation. Thus, the parent class always contains the beginning of the table, and a class that inherits only expands the already existing table if it has new virtual methods, and fills the beginning of the table for all overloaded methods of the parent class. When calling a virtual function is always called root method of the parent class which already places the cursor in the desired function of the offspring, using the virtual function table. A pointer to the virtual function table is always stored inside the object's data (the special field of the class structure). To see more detail as it occurs directly in the code — the output files compiler files with the suffix "_ucxx.cpp".

One of the features of uCxx — is the generation of initialization functions for each module. These functions are used to initialize global objects, populate tables of virtual functions. Calls all functions intialization modules included in the sketch appended inside the setup() function of a custom sketch.

Compiling the entire set of files needed to build the sketch is done twice. On the first pass is determined by many available for call custom methods and a variety of initialization methods in the second pass is generated on the basis of these sets the "updated code", which excludes all unused methods in custom classes. This approach allows to reduce the volume of output in the sketch and while not greatly increasing compile time.

At the final stage for all the resulting "pure" si-file called sdcc, he collects the final hex version of the sketch. That's it — the sketch is ready for download inside Z-Uno

the

Loader

Of course, AVR-DUDE, we too are not perfect. Moreover, we change only the custom part of the code, keeping the Z-Uno our firmware. Because we use the most standard Z-Wave Protocol Serial API, similar to what is used for USB-sticks. It allows you to pass in the Z-Uno sketch (in the auxiliary memory EEPROM), initiate overwriting Code Space (FLASH) and reboot (this work is done by the loader sketch).

For communication on this Protocol with our IOS we wrote our own little tool in Python. She is called to fill in the sketch, as well as new versions of our firmware (bootloader sketch).

the

Libraries and headers

For proper Assembly of custom code we need libraries and files headers for descriptions of the available functions. It is here that described an Arduino-like API. All of this part lies on the Github, it is possible to feel and to rule.

Libraries are often an adaptation of the standard Arduino'vskih libraries to the specifics and architecture of the Z-Uno. Some users have already started to help us, offering pull requests on github with your libraries, or our corrections.

the

operating system Calls, and different ABI

Just to emphasize that the firmware Z-Uno (stack Z-Wave and loader sketch) compiled by the compiler Keil, while a sketch is going to SDCC. To say that the code is incompatible is to say nothing. These compilers use radically different ABI (Application Binary Interface), i.e. notation of passing parameters (via what registers, what order, how to pass a pointer to memory,...) And then we crossed a hedgehog with a snake. To switch from one code to another, we used the idea of a system call in Unix-like OS. Memory was allocated "stack" (in fact just a small sequence of bytes). Both codes know the exact address of this array. Custom code first puts the "syscall number", then the specified procedure put parameters corresponding to this syscall in this array (via zunoPush)after which jumps to the specified address (LCALL) to the code of the loader sketch. The point where the jump is rigidly defined at compile your sketch. Once in the loader code in the sketch, looking at the number of the syscall has already been removed (via zunoPop) settings and runs the desired operation on them. In the opposite direction everything works similarly. The transfer parameters using the "array-stack" allows you to not pay attention to which registers uses a particular compiler (in our case, Keil C51 and SDCC can use different sets of registers).
To make it easier to imagine how different these two compiler understand parameter passing in functions here is a small example. So Keil reports the first single-byte parameter in register R7, and a two-byte parameter in registers R6-R7 (see here), while SDCC this parameter will pass through the DPL in the case of a single-byte parameter, and using the DPL/DPH — in the case of double-byte (see the SDCC manual, p. 53, para "3.12.1 Global Registers used for Parameter Passing"). Thus, there is complete incompatibility of these compilers when passing parameters to functions via registers.

Since both the code (loader sketch /sketch) are compiled separately and about each other do not know anything, they may assume that the registers no one spoils. So we save all the registers during the transition from one code to another and restored when returning back.

What syscall do we have? Well, of course, the implementation of pinMode, digital/analogRead/Write delay (see below), work with Serial0/1, SPI, read/write EEPROM and NZRAM (area XRAM living even during sleep), setting KeyScanner, work with IR-driver in a dream, sending reports and commands to other devices (see ZUNO_FUNC).

the

Stack

First we tried the idea with different stacks and in the transition from the space of Z-Wave in the user and Vice versa. Done by allocating two stack in IDATA and save the SP in the transition. However, this approach was not very economical, because a large nested functions (a C++ attachments a lot) we often overflowed user stack. In General, stack in the 8051 is very limited compared to AVR.

In the end, we came back to the obvious version of the common stack. But there is a caveat. About him below (about the delay).

the

shared memory

In addition to the stack, there are other shared resources. For example, memory. 8051 in its two IRAM and XRAM. Transactions IRAM shorter and faster (MOV), XRAM longer (MOVX). Working with pointers is only possible in XRAM.

In both cases, we just gouged from Keil part of the memory so that its not used, and in SDCC on the contrary only it and let. There is a simple division of resources. Only region to pass parameters to syscall and stack in IRAM is shared (well, of course all the registers is also in IRAM, they also are shared).

the

the Implementation of delay()

Most functions require you to do something and return control fairly quickly. But such a simple function like delay() took a lot of effort. The fact that we can't just lock your chip, making something like while(counter--); as is done in Arduino. If you do so, the radio transmission at that time is interrupted (interrupt radio will work, but not the analysis of the received bytes). And if you delay for longer than 10 MS radio exchange will simply become impossible due to packet loss.

This problem we solved quite tricky: if the delay time less than 10 MS, we leave the loop, which launched a library function work with came radio packages. It is responsible for building the package and transfer into a temporary buffer, the inbound queue. In addition, it implements the relay and other network-level Z-Wave. But you can't do that: the radio control will not work, the answers to the queries, the sensor values will not be sent.

Therefore, when the delay time we have to get out of user code and return to loader code in the sketch, which is responsible for high-level handling of packets and answers to them. In this case, we remember that we are in delay, jump into the bootloader sketch, standard work, but not run of loop(). Once the timer time has passed, and we have to go back, we remove the flag and do a RET to return from delay() in user code.

Pay attention that all the getter and setter still work even while waiting in delay().

the

tires

The tires that are missing hardware drivers (I2C, 1-Wire-specific DHT-11), we have implemented directly in user code, on the basis of GPIO (in libraries, connected to the sketch).

the

Working with pins, fast pins

However, protocols such as I2C may require high speed. To achieve 400 kHz, causing the syscall just will not work. I do a lot of "eats" this level of abstraction. It was, therefore, found another solution. One port (8 pins) we are isolated from the rest and called it a "quick pins". Added a new data type s_pin that at the level of clang (prior to compilation) are converted to a constant, and the functions digitalWrite and digitalRead with such pin is immediately converted to a write to control register pins. For example, to enable P0.5: P05 |= (1 << 5);Also has been added indirect addressing, including pin — when you pass a variable of type myPin s_pin function with digitalWrite or digitalRead with the variable, the latter is converted to direct work with the register. For example, P0 |= (1 << (myPin-9)Note that in the 8051 architecture, cannot be addressed indirectly, any pin, but only within a specific port. That is why we chose a "quick" port P0 (9-16 feet on the Z-Uno). So instead of 1 MS to work with the port via syscall we came to 2 MS for the indirect and 0.5 µs for direct addressing quick pins.

the

That is hidden from the user

Let me remind you, our task was to hide from the user some of the functions as for the NDA, and to simplify. In the end, the whole kitchen is related with Z-Wave is hidden quite — the user is not worried about a lot necessary for compliance with the standard Z-Wave Plus command classes. For example, Associations, firmware updates, setting the time of awakening, a report on the battery life, the test range communications, encryption, channels, version reports device and command classes — and much more already implemented in the correct way. The user have to write the logic of the device connection pins with a custom channel types. For example, when receiving instructions on/Off switch on the first channel to turn on/off the pin, and when receiving instructions on/Off switch on the second channel to send through the UART a command to another microcontroller.

In addition is completely hidden under the hood implementation of the radio part, packet processing and so on, that refers to the standard Z-Wave, and it makes no sense to give the user.

the

Conclusion

In General we managed quite nicely to solve the problem of creating your own Z-Wave devices for people who do not know any details of the Protocol nor of the intricacies of the microcontroller. Simple Arduino experience enough. For the first quarter since the release of Z-Uno we not only managed to sell the planned party, but to build a good community around this project. In addition, we regularly publish new examples of use Z-Uno with various sensors.

By the way, during the work on the project we had two competitors, but both curled up right in front of our launch. It seems that the task was really not easy...

I hope our experience will be useful, and in the comments readers us some intelligent advice.

Article based on information from habrahabr.ru