AudioVision, Assisting vision impaired people

Meeting the brief Investigation Plan & Design Create Evaluation References Summary

A progress log covering the key milestones of the development process

Development Process: My Progress Log

Week Number	Key tasks	Issues discovered
Week 1.	Brainstorming a blind assistance idea and how it would be achieved. Looked at alternative solutions already in place and refined idea to differentiate it. Also developped a method to mount sensors.	Found ideal sensors for the project that are compact, precise and can resolve finer details. But decided to first attempt using ultrasonic sensors because they were more readily available, despite their limitations
Week 2.	Researched how to generate audio programmatically as it is being played. Then i developped code to create the audio dynamically.	Struggled sending the audio player the right data type (float32 instead of float64)
Week 3.	Fixed issue where too many audio streams where being created causing the pi to run out of memory and crash, finally allowing test audio to be created.	Ultrasonic sensors do not read distance at angles over about 20 degrees to a flat surface, aswell as giving unreliable readings off of complex surfaces.
Week 4.	Integrated the distance sensor code with the audio generation and prototyped various warning sounds changing in pitch, tempo and volume.	Noticed a popping sound as the audio doesn't match up from the previous loop.
Week 5.	Added stereo sound capabilities.	Latency is still higher than required and sensors can't detect soft surfaces or sharp angles. Decided to switch to the aformentioned sensors.
Week 6.	Tried to implement example code from github to make the camera recognise objects. Did fresh install of Raspberry pi.	Raspberry pi would crash when the installing required libraries though. Even after attempting to completely wipe the device and try again the problem got even worse when the Raspberry pi would refuse to install any libraries forcing me to revert back to the previous installation.
Week 7.	Changed how audio is prepared so that the waves change smoothly from one frequency to another and the new sample starts in phase with the last reducing latency as well as removing skipping sound. Added exponential fallof for the volume of the audio because it feels more natural and approximates real sound deafening.	I discovered that these sensors cannot have seperate I²C bus addresses meaning they can't work on the same bus and will probably require a multiplexer to use two at a time.
Week 8.	(Mocks)
Week 10.	Got to work trying to make new sensors compatible. Analysed the example arduino code to create an Python equivalent for the raspberry pi.	The registers in the example code were wrong for my application so i had to search them for the usable data. The 2 octet to 16bit integer conversion code was also wrong.
Week 11.	Integrated new sensor code into the main program and also enabled a second IIC bus in the Raspberry pi config file to be able to run left and right sensors simultaneously.	Skipping issue reemerged for no apparent reason after switching to new sensors. Eventually discovered it was due to the program waiting for sensor data to be available after sending request so the audio would pause for a fraction of a second.
Week 12.	Mounted sensors and Raspberry pi to a wrist brace using moldable glue and a needle and thread. Also wired up a button for gathering data to tune the warning response. Added interaural time delay to enhance stereo effect.	Volume L/R shift was switched with the interaural time delay L/R shift leading to a weird effect where I could not figure out where the sound was coming from because these two effects were opposing.
Week 13.	Implemented emergency warning system and realtime graphing. Started recording for video and capturing the sound from the device.	Found strange issue where the audio that was recorded would play on the raspberry pi but not on the laptop, analysing the file in video editing software showed no audio channel. In the end had to play the file specifically in VLC then re-screen record it to capture audio that the video editing software could read.(This introduced a crackle to the audio)

Testing Log

Week Number	Test	Result
Week 3.	Tested Ultrasonic sensor in a variety of situations (different materials, angles, shapes and distances). Compared output with distance measured using a meter stick	Very inconsistent readings on soft surfaces, complex shapes and even on relatively shallow angles.
Week 3.	Tested mono audio generation (pure tone)	Raspberry pi crashes after a few audio tones are played
Week 4.	Tested mono audio generation with 2 harmonics	Audio almost sounds like a heartbeat sounds good as a warning at higher pitch. Audio skips between samples (popping sound).
Week 5.	Tested stereo audio generation	Changing volume in each ear is less noticable than expected. And mismatched tempos between left and right is confusing
Week 7.	Tested audio output changing through a range of frequencies	Audio popping is indeed gone
Week 10.	Tested new infrared LiDar sensor's output against a meter stick	Very precise output down to the millimeter although closest distance it can measure is 2cm

Explain a problem that was encountered in the development of the project and how it was overcome.

My hopes were to simply drop in the new laser ranging sensors and they would work with the simulation and sound generation code but it's never that easy..

I could not find any documentation, guide or even anyone who has connected these sensors to a Raspberry pi. However I did find example code for using the sensors with an arduino that thanks to my previous experience with arduino microcontrollers I might be able to reverse engineer. The code returned no data at first, yet by printing all the registers in the sensor there were some with seemingly random numbers. After researching almost every line of the example code I discovered the & operator which passes the address in memory of a variable instead of it's value in C. This is very strange because the sensor shouldn't be able to access the computers memory and should have no reason to. Despite this the change worked and simply making a variable with a value of 0xB0 in it and sending it's address to the sensor meant the sensor was now active. Looking through all the registers of the sensor i noticed two of them seemed to change by moving the sensor. Through some testing I combined these two bytes of data into a single 16 bit unsigned integer.

And with all this the glorious new sensors sprung to life. They were smaller, quicker, had a better field of view, were more precise and didn't have any of the inconsistencies of the ultrasonic sensors no matter the material or shape.

ㅤ

ㅤ

ㅤ

An explanation of a piece of code that you have designed and created, which was important in the development of the project.

When integrating the sensor distance code with the audio generation I noticed an issue where i heard a popping sound between every audio sample. This only occured when moving the sensor so the distance was changing between samples. After debugging I discovered that the issue was the fact that when the code started generating a new sample it doesn't start where the other one left off. Because the frequency of the wave changes between samples i can't simply change the point in the wave where the new sample starts because it would not match up to the same point in the new frequency. The solution to this was having to calculate the degrees out of phase the new wave would be and find the matching point in the new wave to start from. After adding this compensation value the wave can continue smoothly to the next frequency without a skipping sound.