project

Smart Dashcam for Bicycles – Part 6: Experimenting With A New Camera Platform

Leave a comment

One of the features I have in mind for my bicycle dashcam was license plate recognition. In parts 1, 2 and 3, I experimented with the OpenALPR license plate recognition library and a couple different Pi cameras. I encountered a few challenges:

  • Image quality challenges: out-of-focus images, warped images due to the “rolling shutter” of the Pi camera
  • Field of view: capturing more than just the license plate
  • Speed: Only able to process 1 image every 8 seconds on my Pi 3

I acquired the Luxonis Oak-D AI accelerated camera to experiment with different image sensors which could potentially address my image quality challenges, stereo vision/depth sensing provided interesting capabilities, and the AI acceleration to increase the speed. This spring, I mounted it to my bike and started capturing images on my rides.

I had issues with my Pi 3 – it would stop running reliably after a minute or two – I suspect it had been damaged by vibration from previous rides, being strapped to my bike rack. I acquired a new Pi 4, and was up and running again.

Initially, with the Oak-D setup, I had a lot of the same image quality problems I was having with the Pi 1 and 2 cameras – lots of out-of-focus images, the camera just kept on trying to focus, which is a hard problem with taking photos in moving traffic on a bumpy bicycle ride. My application would also crash – this turned out to be due to filling buffers – I was writing more data to my USB thumb drive than it could handle. I ended up getting acceptable results by reducing my capture speed to 2 fps, recording at 4056×3040, turning auto focus off, locking the focus at its 120 setting, and setting the scene mode sports, in the DepthAI API as follows:

rgb.setFps(2)
rgb.initialControl.setManualFocus(120)
rgb.initialControl.SceneMode(dai.CameraControl.SceneMode.SPORTS) rgb.initialControl.setAutoFocusMode(dai.RawCameraControl.AutoFocusMode.OFF)

With these settings, images are focused in the narrow range where it’s possible to read a license plate – when cars are too far back, the plates are impossible to read anyway, and it doesn’t matter if that’s out of focus. Luxonis will soon launch a model with fixed focus cameras, which should further improve image quality in high vibration environments. I hope to try this out in the future.

I wanted to build a library of images I could later use to test against various machine vision models, and potentially train my own. I posted a the question on the Luxonis Discord channel – their team directed me to their gen2-record-replay code sample. This code allows you to record imagery, and later play it back against a model – it was exactly what I needed. So I started to collect imagery on my next few rides.

computing, project

Lilygo TTGO, TFT_eSPI, and the Dino/T-Rex Game

Leave a comment

Ever since the Espressif’s ESP8266 wi-fi capable microcontroller was launched, I’ve been thinking about all the possibilities for low cost network connected devices. And, nothing world changing, but I have used it to build a data logging CO2 monitor and a device to control my old TV with Alexa.

I have been thinking of NEW possibilities as I see development boards with the ESP8266’s successor, the ESP32, with a small screen, for less than $20CAD shipped from Aliexpress. What can I build with a really tiny internet connected dashboard? So I ordered a Lilygo TTGO.

A day after it arrived, Hackaday published an article about a re-creation of Google Chrome’s T-Rex game for this TTGO dev board. Getting that loaded on to the board seems like a good test. I downloaded the TRexTTGOdisplay and installed Lilygo’s TFT_eSPI driver, compiled and…

undefined reference to `TFT_eSprite::pushToSprite(TFT_eSprite*, int, int, unsigned short)'

Hmm. I search around, and I see a hint in the comment’s of the author’s Youtube video: “You will need to update tft library”. I find the source of the TFT_eSPI library, review it a bit, and see that it is designed for a number of microcontrollers and screen controllers – so I copy the User_Setup_Select.h from the Lilygo repository to Bodmer’s most recent TFT_eSPI libary. For anyone doing this now, this will fix the TFT_eSprite::pushToSprite issue and just work… but I got:

TFT_eSPI/TFT_eSPI.cpp: In member function 'virtual void TFT_eSPI::drawPixel(int32_t, int32_t, uint32_t)':
TFT_eSPI/TFT_eSPI.cpp:3289:21: error: 'SPI_X' was not declared in this scope
while (spi_get_hw(SPI_X)->sr & SPI_SSPSR_BSY_BITS) {};


I take a look at what’s happening around line 3289 in TFT_eSPI.cpp, and it appears to be optimization code for the RP2040 – it shouldn’t be compiled in… Taking a look at line 3285:

// Temporary solution is to include the RP2040 optimised code here
#elif (defined (ARDUINO_ARCH_RP2040) || !defined (ARDUINO_ARCH_MBED)) && !defined(TFT_PARALLEL_8_BIT)

See that exclamation point? And everywhere else in the code there are RP2040 optimizations, I see:

// Temporary solution is to include the RP2040 optimised code here
#elif (defined (ARDUINO_ARCH_RP2040) || defined (ARDUINO_ARCH_MBED)) && !defined(TFT_PARALLEL_8_BIT)

Cool, I’ll submit a patch. So I fork the code, and… I don’t see the bug, it’s already been fixed.

I was hoping for another successful contribution to open source, but I was beaten to the punch – if I had started this project a day later, it just would have worked with the latest TFT_eSPI library. In any case, the important thing is, I got TRexTTGOdisplay running. My next project for this dev board will be a little internet connected dashboard.

    project

    Smart Dashcam for Bicycles – Part 5: Blindspot Detection

    1 Comment

    I continue to experiment with how a dashcam can assist urban cyclists. This time, I’ve started a fresh design with a different idea, a new camera, new models, and new code, which I’m submitting as an entry for the Toronto ♥️’s Bikes Make-a-Thon.

    I enjoy biking from my home in North York, near Mel Lastman Square, to my office near Union Station during the week. The most harrowing part of this ride is the Yonge-401 interchange, which requires two lane changes with fast moving traffic from the 401 on and off ramp.

    Yonge - Hwy 401 Interchange
    Yonge – Hwy 401 Interchange

    As cyclists in the city, we all have “scary spots” like these on our routes. I would like to present you with a Smart Dashcam for Bicycles as a tool for these challenges. A dashcam could:

    • increase safety
    • collect evidence in the event of an incident
    • gather data

    For the purposes of the Make-a-Thon, I have built a smart dashcam with blind spot detection, similar to what you would see in modern cars. The IIHS says that in cars, this feature lowers the rate of all lane-change crashes by 14 percent.

    My prototype consists of a laptop, a USB AI accelerated camera from Luxonis mounted to my bicycle seat post, and my smartphone as a display. It’s a few hundred lines of Python code that builds on a freely available AI vehicle recognition model from the Intel Open Model Zoo. I’ve built on the license plate recognition and MJPEG video streaming sample code from Luxonis that was supplied with the OAK-D camera. I tether the laptop to the smartphone using wifi, and I use an iOS app called IPCams to view the video stream.

    Bicycle Dashcam with Smartphone Display
    Bicycle Dashcam with Smartphone Display

    The vehicles are recognized and identified. The video is streamed over wifi to the smartphone. A caution alert is added to the video when a vehicle is detected.

    Phone screen shot. Caution, car approaching
    Phone screen shot. Caution, car approaching

    A demo video can be found here: https://youtu.be/zMTRDsA6uJM

    In this proof of concept, the dashcam is just a fancy, complicated, expensive, rear view mirror. A final version would expand on this functionality by integrating features such as:

    • Sounding an audible alert when danger is detected
    • Recording the speed and proximity of the cars around you
    • Integrated GPS
    • Cloud and social features for sharing data with the city and fellow cyclists
    • A car driver readable display, eg: “Driver ABCD1234, your current speed is 45”. Like a mobile Toronto Watch Your Speed program sign. Would a driver allow a cyclist more space if they were aware their actions are being logged?

    Parts:
    Luxonis OAK-D Camera
    Bike Phone Mount
    Bike Camera Mount
    IPCams for iOS (to watch MJPEG stream)

    Source:
    https://github.com/raudette/SmartDashcamForBikes

    Previous Articles:
    See my previous articles on bicycle dashcams:
    Part 4, Part 3, Part 2, Part 1

    Updates


    This project was featured on Hackaday, November 1, 2021.

    Follow-up articles:
    Part 6, Part 7

    New Source Code Base:
    I am exploring a different direction.
    https://github.com/raudette/SmartDashcamForBikesMk3

    computing, security, thoughts

    Mesh network in my neighborhood. Not how I had envisioned it.

    Leave a comment

    I got an email from Amazon today. I’m automatically opted-in to “Amazon Sidewalk”, unless I choose to opt out. Amazon Sidewalk allows devices participating their Sidewalk program to connect to the Internet through Amazon devices, like the Amazon Echo.

    Amazon Sidewalk – a mesh network for Amazon devices to connect to the Internet:
    https://www.amazon.com/Amazon-Sidewalk/b?ie=UTF8&node=21328123011

    Not exactly how I’d envisioned a neighborhood mesh, but their “read the fine print to opt out” strategy will probably work better than my asking neighbors to build a network.

    computing, project

    1-Click Passwords

    Leave a comment

    I was recently presented with a situation where I would have to regularly enter a 48 random character password for a month or more to log in to a computer that was assigned to me. Given that I couldn’t possibly memorize this string, and the computer is reasonably physically secure, I decided to build a device to do this for me.

    I had previously used an Arduino to emulate a gamepad for a homemade Dance Dance Revolution mat. This time, I needed to emulate a keyboard. A search for “HID Arduino” returned the Arduino HID page, which suggested an Arduino with an Atmel 32u4 microcontroller. A search for Arduino 32u4 on Amazon returned the KeeYees Pro Micro clone, which I ordered.

    Arduino Pro Micro Clone, button wired to I/O 4

    It came in, I soldered a button to I/O 4, and uploaded the following code:

    include "Keyboard.h"
    include "Bounce2.h"
    const int buttonPin = 4;
    Bounce bounceTrigger = Bounce();
    void setup() {
    bounceTrigger.attach(buttonPin, INPUT_PULLUP );
    Keyboard.begin();
    }
    void loop() {
    bounceTrigger.update();
    if ( bounceTrigger.rose() ) {
    Keyboard.println("I put my password here");
    }
    }

    Now, every morning, instead of copying 48 characters from a Post-it, I just click the button.

    It should be said, this defeats the purpose of the password, and the password isn’t stored in a secure way on the microcontroller. But this technique is great for any time you need to automate a sequence of keystrokes.

    project

    Bicycle Dashcam Part 4: New Hardware

    Leave a comment

    I was reading an article about Oak Vision Modules on Hackaday, and thought, wow, this is the PERFECT platform for my bicycle dashcam. The Oak Vision module is a Kickstarter project with camera modules, depth mapping capability using stereo vision, and a processor (Intel Movidius Myriad X) designed to accelerate machine vision in 1 package for $149US – see https://www.kickstarter.com/projects/opencv/opencv-ai-kit/

    At the 3:55 mark in the marketing video, I THEN see the board mounted to a bicycle saddle, which is EXACTLY what I want to do:

    Luxonis OAK-D vision module prototype as seen in their Kickstarter video

    I went to see what I could find about the developers, and read about them on TechCrunch:
    “The actual device and onboard AI were created by Luxonis, which previously created the CommuteGuardian, a sort of smart brake light for bikes that tracks objects in real time so it can warn the rider. The team couldn’t find any hardware that fit the bill so they made their own, and then collaborated with OpenCV to make the OAK series as a follow-up.”

    This is pretty exciting – CommuteGuardian is the first project I’ve come across with similar goals to mine: Prevent and Deter Car-Bicycle accidents. I exchanged a few emails with Brandon Gilles, the Luxonis CEO, and he shared some background – they also checked out OpenALPR, and started work on mobile phone implementations, but decided to move to a custom board when the Myriad X processor was launched.

    You can read more about CommuteGuardian here:
    https://luxonis.com/commuteguardian and
    https://discuss.luxonis.com/d/8-it-works-working-prototype-of-commute-guardian

    I decided to back Luxonis’ Oak project. I’ll have to learn some new tools, but this board will be much faster than the Pi for image analysis (much faster than the 1 frame per 8 seconds I’m getting now!). The stereo vision capabilities on the Oak-D will allow for depth mapping, a capability for which I had previously been considering adding a LIDAR sensor. Looking forward to receiving my Oak-D, hopefully in December. In the interim, I’ll continue to experiment with different license plate recognition systems, read more about the tooling I can use with Oak-D, and perhaps try a different camera module on the Pi.

    project

    Bicycle Dashcam Part 3: More field testing

    Leave a comment

    On a sunny mid-June Saturday, I took my bike for a ride down Yonge St to lake Ontario with my bicycle dashcam, testing my latest changes (May 18th). Over the course of a 2 hour ride, taking a photo about every 10 seconds:

    • Reviewing the photos with my own eyes, I can make out about 45 images with readable plates (not every image was usable or had a car in the photo)
    • Of these 45, OpenALPR can make out about 10
    Sign misidentified as license plate
    OpenALPR picked out this dry cleaning sign as license plate 2 H0UR

    I’m going to try running these photos through alternate ALPR engines, and compare results.

    On this run, I tested the Pi Camera V2’s various sensor modes: the streaming modes at 1920×1080 30 fps, 3280×2464 15 fps, 640×922 30 fps, 1640×922 40 fps, 1280×720 41 fps, 1280×720 60 fps, as well as the still mode at 3280×2464. Further testing is likely still required, but I continue to get the best results from the still mode – all of the successful matches were shot using still mode.

    Dashcam Successfully Recognizes Plate
    Dashcam Successfully Recognizes License Plate

    I’m getting better results than I had on previous runs as a result of tweaking the pi-camera-connect NodeJS library to:

    • use a 5 second capture delay, which allows exposure time, gain, and white balance to be determined
    • set the exposure mode to sports, which reduces motion blur by preferentially increasing gain rather than exposure time

    However, the images are still not as good as I would like.

    Rolling shutter issues are apparent in some photos taken by the Pi Camera V2 while in motion

    The Plate Recognizer service has an excellent article on Camera Setup for ALPR. It highlights many of the challenges I’m seeing with my setup:

    • Angle, lighting
    • 8 MP is suggested for highway or street monitoring – the Pi Camera is sufficient in this regard
    • Zoom – I think this is a challenge in my setup – I liked the idea of getting everything around me, but I think I have to reduce the area I capture to get a view of the plate with more detail. Perhaps focus to my “7 o’clock” rather than capture everything behind me.
    • At 30 mph (~45 km/hr), which probably covers most bike riding in traffic, they suggest at least 3 to 5 frames at 15-25 frames per second.

    I might order the latest Pi camera with the zoom lens and see if I get better results.

    Reviewing the data from my ride, there’s also an issue with my code that pulls the GPS coordinates from the phone, which I didn’t see when testing at home. I figure this is either the phone locking while I ride, and not running the javascript – I’ll try using the NoSleep.js library before my next test run.

    My personal brain dump, Opinions, Projects, Toronto