All posts by raudette

Smart Dashcam for Bicycles – Part 7: Training A Vision Model

I wanted to build my own vision model for a few reasons:

  1. I wanted to learn how
  2. In my limited experience with OpenALPR, it looked like it was missing some license plates that seemed fairly readable to my eyes – could I possibly do better training my own model?
  3. Just the way it is built, I know I wouldn’t be able to get OpenALPR to run faster on my Pi – I wouldn’t be able to get it to run faster by off loading image processing to a VPU like the Myriad X in my Oak D camera.
  4. The gen2-license-plate-recognition example provided by Luxonis, built from Intel’s Model Zoo, does not work well with Ontario license plates

The first step was building a library of images to train a model with. I sorted through hundreds of images I’d taken on rides in September, and selected 65 where the photos were clear, and there were license plates in the frame. As this was my first attempt, I wasn’t going to worry about sub-optimal situations (out of focus, low light, over exposed, etc…). I then had to annotate the images – draw boxes around the license plates in the photos, and “tag” them as plates. I looked at a couple tools – I started with Microsoft’s VoTT, but ended up using labelimg. Labelimg was efficient, with great keyboard shortcuts, and used the mouse scroll wheel to control zoom, which was great for labeling small details in larger photos.

I then tried one tutorial after another, and struggled to get them to work. Many examples were setup to run on Google Colab. I found when I was following these instructions, and I got to part where I was actually training the model, Colab would time out. Colab is only intended for short interactive sessions – perhaps it wouldn’t work for me as I was working with higher resolution images, which would take more computing time.

What I ended up doing was manually running the steps in the Train_YoloV3.ipynb notebook from pysource, straight into the console. As my home PCs don’t have dedicated GPUs, I setup a p3.2xlarge Amazon EC2 instance to run the training. If memory serves, training against those 65 images, using the settings from that tutorial, took a couple of hours.

I took the model I created from my September rides, and then tested it against images from my October rides – I’m surprised how well it worked.

My Yolov3 model running on Oak-D

Since training that model, I’ve been on the lookout for an nVidia video card I can use for training at home. It’s hard to know for sure, but it seems it wouldn’t take long to recoup the cost of a GPU vs training on an EC2 instance in the cloud, and I can always resell a GPU. I’ve tried a few times with the fastest CPU I have in the house (a Ryzen 3400g), and it just doesn’t seem feasible. I haven’t seen a cheap GPU option, and the prices just seem to be going higher since I started looking in November.

I don’t have usable code or a useful model to share at this point, at this point, I’m mostly learning and trying to figure out the process.

Printing and Binding an ePub eBook

I wanted a hard copy of an eBook I had that is out of print. There are many resources out there for binding books. Many recommend using acid free PVA glue. I can’t speak to how it compares to other glues, but “Aleene’s Tacky Glue” is a PVA glue, available acid free, which was available at craft stores in my area.

This post will focus on prepping an eBook for print. As US Letter is the common paper size here, which is too big for a book, I decided to print 4 pages per US letter page, 2 pages per side, each 5.5″ wide by 8.5″ tall.

First, I loaded to book into Calibre, opened it, and printed it to PDF. For this exercise, I’ve used Ian Fleming’s Casino Royale, which is out of copyright in Canada.

Calibre Print to PDF screen
Calibre Print to PDF

Next, I had to re-arrange the pages. If we just print 2 pages per side, duplex, page 4 will end up on the back of page 1. We want page 2 on the back of page 1 – we want to reorder the PDF following the patterns 1, 3, 4, 2, 5, 7, 8, 6… This LibreOffice spreadsheet might help: Pages.ods

Illustration of required page ordering
Pages have to be re-ordered for regular duplex printing – page 2 has to be on the back of page!

PDFTK is a great tool for re-ordering PDFs. I have re-ordered the book, skipping the first page, with PDFTK as follows:
pdftk Casino\ Royale.pdf cat 2 4 5 3 6 8 9 7 10 12 13 11 14 16 17 15 18 20 21 19 22 24 25 23 26 28 29 27 30 32 33 31 34 36 37 35 38 40 41 39 42 44 45 43 46 48 49 47 50 52 53 51 54 56 57 55 58 60 61 59 62 64 65 63 66 68 69 67 70 72 73 71 74 76 77 75 78 80 81 79 82 84 85 83 86 88 89 87 90 92 93 91 94 96 97 95 98 100 101 99 102 104 105 103 106 108 109 107 110 112 113 111 114 116 117 115 118 120 121 119 122 124 125 123 126 128 129 127 130 132 133 131 134 136 137 135 138 140 141 139 142 144 145 143 146 148 149 147 150 152 153 151 154 156 output collated.pdf

Next, I used a tool called pdfjam to fit 2 pages per side:
pdfjam collated.pdf -o collated-2perpagealternate.pdf --nup 2x1 --landscape

I sent this PDF to my local printer, and had them cut the pages in half for me. With this output, I bound the book, roughly following a Youtube tutorial. My book turned out OK, but it feels like it would take me a few more attempts to get a book as sturdy as a commercially bound book.

Smart Dashcam for Bicycles – Part 6: Experimenting With A New Camera Platform

One of the features I have in mind for my bicycle dashcam was license plate recognition. In parts 1, 2 and 3, I experimented with the OpenALPR license plate recognition library and a couple different Pi cameras. I encountered a few challenges:

  • Image quality challenges: out-of-focus images, warped images due to the “rolling shutter” of the Pi camera
  • Field of view: capturing more than just the license plate
  • Speed: Only able to process 1 image every 8 seconds on my Pi 3

I acquired the Luxonis Oak-D AI accelerated camera to experiment with different image sensors which could potentially address my image quality challenges, stereo vision/depth sensing provided interesting capabilities, and the AI acceleration to increase the speed. This spring, I mounted it to my bike and started capturing images on my rides.

I had issues with my Pi 3 – it would stop running reliably after a minute or two – I suspect it had been damaged by vibration from previous rides, being strapped to my bike rack. I acquired a new Pi 4, and was up and running again.

Initially, with the Oak-D setup, I had a lot of the same image quality problems I was having with the Pi 1 and 2 cameras – lots of out-of-focus images, the camera just kept on trying to focus, which is a hard problem with taking photos in moving traffic on a bumpy bicycle ride. My application would also crash – this turned out to be due to filling buffers – I was writing more data to my USB thumb drive than it could handle. I ended up getting acceptable results by reducing my capture speed to 2 fps, recording at 4056×3040, turning auto focus off, locking the focus at its 120 setting, and setting the scene mode sports, in the DepthAI API as follows:

rgb.setFps(2)
rgb.initialControl.setManualFocus(120)
rgb.initialControl.SceneMode(dai.CameraControl.SceneMode.SPORTS) rgb.initialControl.setAutoFocusMode(dai.RawCameraControl.AutoFocusMode.OFF)

With these settings, images are focused in the narrow range where it’s possible to read a license plate – when cars are too far back, the plates are impossible to read anyway, and it doesn’t matter if that’s out of focus. Luxonis will soon launch a model with fixed focus cameras, which should further improve image quality in high vibration environments. I hope to try this out in the future.

I wanted to build a library of images I could later use to test against various machine vision models, and potentially train my own. I posted a the question on the Luxonis Discord channel – their team directed me to their gen2-record-replay code sample. This code allows you to record imagery, and later play it back against a model – it was exactly what I needed. So I started to collect imagery on my next few rides.

Lilygo TTGO, TFT_eSPI, and the Dino/T-Rex Game

Ever since the Espressif’s ESP8266 wi-fi capable microcontroller was launched, I’ve been thinking about all the possibilities for low cost network connected devices. And, nothing world changing, but I have used it to build a data logging CO2 monitor and a device to control my old TV with Alexa.

I have been thinking of NEW possibilities as I see development boards with the ESP8266’s successor, the ESP32, with a small screen, for less than $20CAD shipped from Aliexpress. What can I build with a really tiny internet connected dashboard? So I ordered a Lilygo TTGO.

A day after it arrived, Hackaday published an article about a re-creation of Google Chrome’s T-Rex game for this TTGO dev board. Getting that loaded on to the board seems like a good test. I downloaded the TRexTTGOdisplay and installed Lilygo’s TFT_eSPI driver, compiled and…

undefined reference to `TFT_eSprite::pushToSprite(TFT_eSprite*, int, int, unsigned short)'

Hmm. I search around, and I see a hint in the comment’s of the author’s Youtube video: “You will need to update tft library”. I find the source of the TFT_eSPI library, review it a bit, and see that it is designed for a number of microcontrollers and screen controllers – so I copy the User_Setup_Select.h from the Lilygo repository to Bodmer’s most recent TFT_eSPI libary. For anyone doing this now, this will fix the TFT_eSprite::pushToSprite issue and just work… but I got:

TFT_eSPI/TFT_eSPI.cpp: In member function 'virtual void TFT_eSPI::drawPixel(int32_t, int32_t, uint32_t)':
TFT_eSPI/TFT_eSPI.cpp:3289:21: error: 'SPI_X' was not declared in this scope
while (spi_get_hw(SPI_X)->sr & SPI_SSPSR_BSY_BITS) {};


I take a look at what’s happening around line 3289 in TFT_eSPI.cpp, and it appears to be optimization code for the RP2040 – it shouldn’t be compiled in… Taking a look at line 3285:

// Temporary solution is to include the RP2040 optimised code here
#elif (defined (ARDUINO_ARCH_RP2040) || !defined (ARDUINO_ARCH_MBED)) && !defined(TFT_PARALLEL_8_BIT)

See that exclamation point? And everywhere else in the code there are RP2040 optimizations, I see:

// Temporary solution is to include the RP2040 optimised code here
#elif (defined (ARDUINO_ARCH_RP2040) || defined (ARDUINO_ARCH_MBED)) && !defined(TFT_PARALLEL_8_BIT)

Cool, I’ll submit a patch. So I fork the code, and… I don’t see the bug, it’s already been fixed.

I was hoping for another successful contribution to open source, but I was beaten to the punch – if I had started this project a day later, it just would have worked with the latest TFT_eSPI library. In any case, the important thing is, I got TRexTTGOdisplay running. My next project for this dev board will be a little internet connected dashboard.

Smart Dashcam for Bicycles – Part 5: Blindspot Detection

I continue to experiment with how a dashcam can assist urban cyclists. This time, I’ve started a fresh design with a different idea, a new camera, new models, and new code, which I’m submitting as an entry for the Toronto ♥️’s Bikes Make-a-Thon.

I enjoy biking from my home in North York, near Mel Lastman Square, to my office near Union Station during the week. The most harrowing part of this ride is the Yonge-401 interchange, which requires two lane changes with fast moving traffic from the 401 on and off ramp.

Yonge - Hwy 401 Interchange
Yonge – Hwy 401 Interchange

As cyclists in the city, we all have “scary spots” like these on our routes. I would like to present you with a Smart Dashcam for Bicycles as a tool for these challenges. A dashcam could:

  • increase safety
  • collect evidence in the event of an incident
  • gather data

For the purposes of the Make-a-Thon, I have built a smart dashcam with blind spot detection, similar to what you would see in modern cars. The IIHS says that in cars, this feature lowers the rate of all lane-change crashes by 14 percent.

My prototype consists of a laptop, a USB AI accelerated camera from Luxonis mounted to my bicycle seat post, and my smartphone as a display. It’s a few hundred lines of Python code that builds on a freely available AI vehicle recognition model from the Intel Open Model Zoo. I’ve built on the license plate recognition and MJPEG video streaming sample code from Luxonis that was supplied with the OAK-D camera. I tether the laptop to the smartphone using wifi, and I use an iOS app called IPCams to view the video stream.

Bicycle Dashcam with Smartphone Display
Bicycle Dashcam with Smartphone Display

The vehicles are recognized and identified. The video is streamed over wifi to the smartphone. A caution alert is added to the video when a vehicle is detected.

Phone screen shot.  Caution, car approaching
Phone screen shot. Caution, car approaching

A demo video can be found here: https://youtu.be/zMTRDsA6uJM

In this proof of concept, the dashcam is just a fancy, complicated, expensive, rear view mirror. A final version would expand on this functionality by integrating features such as:

  • Sounding an audible alert when danger is detected
  • Recording the speed and proximity of the cars around you
  • Integrated GPS
  • Cloud and social features for sharing data with the city and fellow cyclists
  • A car driver readable display, eg: “Driver ABCD1234, your current speed is 45”. Like a mobile Toronto Watch Your Speed program sign. Would a driver allow a cyclist more space if they were aware their actions are being logged?

Parts:
Luxonis OAK-D Camera

Bike Phone Mount
Bike Camera Mount
IPCams for iOS (to watch MJPEG stream)

Source:
https://github.com/raudette/SmartDashcamForBikes

Previous Articles:
See my previous articles on bicycle dashcams:
Part 4, Part 3, Part 2, Part 1

Updates


This project was featured on Hackaday, November 1, 2021.

Follow-up articles:
Part 6, Part 7

New Source Code Base:
I am exploring a different direction.
https://github.com/raudette/SmartDashcamForBikesMk3

Mesh network in my neighborhood. Not how I had envisioned it.

I got an email from Amazon today. I’m automatically opted-in to “Amazon Sidewalk”, unless I choose to opt out. Amazon Sidewalk allows devices participating their Sidewalk program to connect to the Internet through Amazon devices, like the Amazon Echo.

Amazon Sidewalk – a mesh network for Amazon devices to connect to the Internet:
https://www.amazon.com/Amazon-Sidewalk/b?ie=UTF8&node=21328123011

Not exactly how I’d envisioned a neighborhood mesh, but their “read the fine print to opt out” strategy will probably work better than my asking neighbors to build a network.

1-Click Passwords

I was recently presented with a situation where I would have to regularly enter a 48 random character password for a month or more to log in to a computer that was assigned to me. Given that I couldn’t possibly memorize this string, and the computer is reasonably physically secure, I decided to build a device to do this for me.

I had previously used an Arduino to emulate a gamepad for a homemade Dance Dance Revolution mat. This time, I needed to emulate a keyboard. A search for “HID Arduino” returned the Arduino HID page, which suggested an Arduino with an Atmel 32u4 microcontroller. A search for Arduino 32u4 on Amazon returned the KeeYees Pro Micro clone, which I ordered.

Arduino Pro Micro Clone, button wired to I/O 4

It came in, I soldered a button to I/O 4, and uploaded the following code:

include "Keyboard.h"
include "Bounce2.h"
const int buttonPin = 4;
Bounce bounceTrigger = Bounce();
void setup() {
bounceTrigger.attach(buttonPin, INPUT_PULLUP );
Keyboard.begin();
}
void loop() {
bounceTrigger.update();
if ( bounceTrigger.rose() ) {
Keyboard.println("I put my password here");
}
}

Now, every morning, instead of copying 48 characters from a Post-it, I just click the button.

It should be said, this defeats the purpose of the password, and the password isn’t stored in a secure way on the microcontroller. But this technique is great for any time you need to automate a sequence of keystrokes.