Arduino Serial transfer protocol

[Rx7man]

I'm working on a turbocharger controller that has to do a lot of real-time (well, close to it) work, it's powered by an Arduino Mega 2560, which has lots of memory, but is quite stingy on processing power.. and to make it harder, transfer has to be done over a serial port to another unit (another Arduino, Raspberry pi, or laptop) to display the current parameters.
It will also have to take data back from whatever unit is connected to it such as tuning parameters and apply them to the program or save to EEPROM.

The data will be a mix of types.. integer, floats, signed, unsigned, and possibly arrays of them or custom data types as well.

I'm looking for guidelines on how to set this up... Currently, I'm looking at a 'packet' that looks somewhat like this

Start marker, 1 byte, probably 0xFF or 0xFE
Sender Address, 1 byte (optional)
Destination address, 1 byte, 0xFF for broadcast (optional)
Packet length, 1 byte (will limit packet size to 255 bytes, yes, I'm OK with that)
Data XXX bytes
Checksum 1 byte (will be a simple one)
Finish marker, 1 byte, probably be 0xFF

That will take care of the sending and receiving of the data, but I will need to do post-processing on it, so the data shown above will contain other things..
I'm thinking of setting it up like this

Data type = Byte 1, indicates the type of data that follows,.. integer, float, signed, or custom.
Destination = Byte 2,3... Where or what do do with this packet of data

I am thinking of using a 'union' to get the bitwise representation of floating point numbers and other types to reduce string parsing.
Arrays will need an additional byte to indicate the length, otherwise they'll be similar
Strings will be handled as character arrays, so no change there except.
I will probably have some method of invoking functions and commands.. such as 'save to eeprom', Reset, Recalibrate, etc.. That can be a data type as well with it's own handlerIs there anything glaring that I'm missing.. perhaps something I should consider, or reconsider?

I enjoy learning the logic of doing it, I'm not very good at C flavored programming, but have moderate knowledge on VB.. I have tried very hard to learn how to be OO, and am starting to 'get it'. I like writing my own code, even if it might not always be the best available, I'll learn how to make it better over time.

 

[jedishrfu]

Depending on the size of your data you may need a larger checksum.

Do you have to deal with big/little endian formatting for your binary numbers?

https://en.wikipedia.org/wiki/Endianness

I would expand your single byte header values to 2 byte numbers.

As an example the start byte could be the CAFE or FACE which can help with endianness determinations.

There's a common notion in comp sci called "Reader make right" where the reader of data first handles reformatting the data if needed.

Do you need a msg# or something to tell you if you've lost a message?

or are you sending a request and expecting a response?

Because you're using an end marker of 0xFF then you have to be careful that you don't interpret your binary data bytes ie there may be an embedded 0xFF that you might think has ended your message or you'll need to exclude it from your data and that gets into other problems of how to represent byte codes that aren't allowed.

 

[Svein]

There are several protocols already defined that you should look at (why re-invent something?)

• SLIP (https://en.wikipedia.org/wiki/Serial_Line_Internet_Protocol). Defines a framework, but leaves the rest to you
  
• ProfiBus (https://en.wikipedia.org/wiki/Profibus). Defines almost everything about data transfer between a device and a controller
  
• PPP (https://en.wikipedia.org/wiki/Point-to-Point_Protocol). A more stable (and complicated) version of SLIP

 

[jedishrfu]

I once worked on a project for an independent study of Real time programming. We planned to link two ATARI 800's together. At the time, the only way was via the serial port. The idea was to make a jet fighter game similar in concept to the movie Firefox where Clint Eastwood stole a plane that was controlled by thinking in Russian. We called our game Fighter Fox and said nyet to the Russian part.

The communication issues we ran into were:

1) At startup who sends the first message:

One approach was to define a start sequence ie computer A is booted sends a message and listens and then computer B is booted, sends a message and listens.

Another approach is for both of them to sit and listen and then have the user interact with one of the computers that in turn causes it to send something to the other one.

2) Structure of your message, we had two identical computers that would share positioning info back and forth over the serial port.

We chose a simple fixed message that sent a latitude, longitude, velocity vector over. It was a fixed byte message that was sent with all data as binary integers.

3) We had a hardware limitation where the sound chip was intimately connected of the comm port and as a result making a sound screwed up the message we were sending.

Our solution was to forego the sound as it was really a frill. We made up for it using beatbox like sounds that we made ourselves (woosh, boom...)

Anyway, if you can implement a known simple protocol that would be best. Also you may have to consider sending data as ASCII number strings so you have a simple means of debugging your code or not your choice.

 

[Rx7man]

Jed, in this case, I'm thinking of not looking for the Message End marker until Message Length bytes have been received, that would allow any combination of bytes to be sent.. Once you've received the full length message, the next byte would be the checksum, and then the frame end... That would mean a false Start of Message would be incredibly unlikely to have a correct checksum and a correct End of Message marker... I should also be able to have a mixed-mode, where I can transfer plain text between messages (debugging?)

Thanks for the link on SLIP/PPP, I don't think that matches what I need very well, but Profibus looks promising, if just for ideas

I plan on forcing all transfers to MSB first since when using Unions in Arduino C, that's the way they end up, and if I don't need to check for it, I can save that processing overhead.. On my VB developed software, I'll use the BitConverter class, and it does it the other way around, but a desktop has a lot more processing power to do the conversion

I did think of sending strings of numeric digits, (just for readability), but then I have to do extensive parsing on the other end, have a much longer message, and in many cases with floating point numbers, have a loss of accuracy.

I'll have to remember to program some Whoosh and Boom sounds in there! :)

 

[jedishrfu]

As alternative to ASCII strings of numbers, you could consider packed decimal representation again for debugging:

https://en.wikipedia.org/wiki/Binary-coded_decimal

Basically one byte = 2 decimal digits and there's 4 bit patterns for +, - and the decimal pt.

 

[Rx7man]

I didn't consider BCD, though I think the extra overhead involved in conversion would be hard to justify, since you still wouldn't really have human readable output. I'll think about it some more and research it more.