This page is a chronologial documentation of my final project for the Living With Robots class at NYU 2020 Spring.
March 31
Idea
My idea is to make a wearable device to enhance music listening experience at home. I love music. I listen to music at home, and I often go to live concerts. All this time, I've been thinking about the difference between going to a venue and listening to a concert, and listening to a broadcasted live concert at home. Meanwhile, the coronavirus situation has made it difficult to hold live concerts at all. I would like to take this as a good opportunity to work on developing a device that will enhance the music experience at home.
There are many possible differences between going to a concert and watching it live at home; quality of the sound, loudness of the sound, smell, lighting, sense of sharing a certain space/time/experience with musicians and other audiences, etc.
In this project, I'm going to focus on the vibration. Human beings experience music not only with their ears, but also with their bodies. Depending on the frequency of the sound, the lower parts of the body (such as the belly and legs) are feeling the vibration in the low notes and the upper parts of the body (such as the skull and chest) in the high notes. My hypothesis is, can it ehnance music listening experience at home by recreating such vibrations?
I haven't decided on a final form, but I'm envisioning either an outfit (jacket, trousers, jumpsuit, etc.) or something with bands wrapped around a few places on the body. I'm also considering using speakers or vibrating motors.
References
1st Iteration
I used one vibration motor and 3V button battery. Placing the motor under my waist, I lay down and manually control the vibration according to the music.
Here are what I felt:
-
I only used one motor, the motor was too small, and the vibration was so focused on one point that I was not able to find the vibration with my whole body.
-
It might be possible to spread the vibrations by attaching some kind of device to the motor or covering the motor.
-
It may also improve the experience by increasing the number of motors and feeling the vibrations in multiple locations of the body.
April 4
Progress
-
Found a paper of LIVEJACKET: Wearable Music Experience Device with Multiple Speakers. This is a great resourse for me!
-
Did some research and ordered some things from amazon such as piezo speakers, batteries, vibration motors, etc.
2ne Iteration Idea
- For the next iteretion, I am going to use PCA9685 with raspberry pi to control 16 vibration motors. In the past, I have used 16 PCA9685s to control 16*16=256 servo motors. I am not sure yet PCA9685 is compatible with vibration motors but it should be fine.
April 8
Progress for 2nd Iteration
-
Connect raspberry pi with PCA9685: Hooking it Up
-
Do configuration: Configuring Your Pi for I2C
-
Install necessary library: Using the Adafruit Library
-
Simple LED blink test: Dimming LEDs
Works good!
April 9
Progress for 2nd Iteration
The problem is how to power a vibration motor with PCA9685? Simply replacing a LED to a vibration motor with external 3V power connected to PCA9685 did not work. It is probably because of the fact that “PCA9685 will limit the current to around 10mA”? (see Dimming LEDs).
It seems that I also need to learn something more about Pulse Width Modulation (PWM) signal and vibration motor. Not sure about them yet but the thing I know for now is PCA9685 and related Adafruit libraries are for controlling PWM signal to power a servo motor (not a DC motor but the motor powered by PWM signal), and you need some kind of trick to power a vibration motor (DC motor) with them.
I am googling to get more information for my learning step.
April 10
Progress for 2nd Iteration
Findings
Here are my findings after some google search:
“The pins on the PCA9685 can only source 10mA or sink 25mA” (16-Channel Servo Driver - PCA9685), meaning the current shold be amplified to run vinbration motors. Also, be sure that PCA9685 max voltage input is 5.5V (PCA9685 datasheet).
There seems at least two workarounds.
(1) Use H-Bridge with PCA9685 Some refs:
(2) Use transistor with PCA9685
Did not sure about transistor but it seems something to amplify current. (How a transistor amplifies current | Digital electronics (6 of 10))
Also, ULN2803A (transistor arrays) seems a good choice for me because I want to run many vibration motors. (Hooking up 0-10V PWM with PCA9685 and Darlington)
Things to buy
So I am ordering some L298N and ULN2803A to see how they work.
Refs
Some other references:
-
Driving Vibration Motors With Pulse Width Modulation (PWM) – AB-012 Released
-
HapticToolkit: easily integrate and control vibration motor arrays for wearables
April 12
Progress for 2nd Iteration
Findings
By the way, I forgot to measure the necessary voltage and current for powering a vibration motor, so I did it with multi meter. Using 3V button battery (was 2.8V measured with multi meter) connected to a vibration motor, the current was about 40mA and voltage drop was about 2.2V.
April 13
Progress for 2nd Iteration
I was able to power 2 vibration motors via PWM with PCA9685 and ULN2803A.
This makes me decide to use RaspberryPi, PCA9685, and ULN2803A to run multiple vibration morots.
I will connect more vibration motors, attach them to various parts of my body, and see what the experience will be like.
April 14
Progress for 2nd Iteration
I did some wiring and soldering for vibration morots.
For the test, I hooked up eight motors to see if they worked. However, when I connect more than 5 motors, it does not work correctly. I am assuming this relates to low voltage because I am connecting one 3V button battery and motors in parallel.
ADDITIONAL NOTE Changing to USB 5V from home socket works.
April 15
Progress for 2nd Iteration
Keep doing wiring. I got a heatgun.
April 16
Progress for 2nd Iteration
After the class and my personal experience, I changed my idea from making a wearble vibration clothe to cushion. The main reason is that when I was frustrated about something personal, I found myself gradually relaxing after cuddling my favorite stuffed animal and falling asleep. So, I decided to combine this phenomenon with the idea of healing robots I had learned in class.
I didn't want to damage my favorite stuffed animal, so for 2nd iteration, I decided to use a cushion.
Today, I cut a fabric into my cushion size piece and stuck eight vibration motors on that it.
Each vibration motor is secured to the cloth using kitchen paper and staples.I know that this method is problematic in terms of strength, but I think it's useful as a prototype because it's easy and can be implemented quickly.
April 17
2nd Iteration Completed
I use jumper wires for the temporary wiring.
Put the cover on.
OK! There are 8 vibration morots in the cushion.
It looks like this.
Here is a testing video. I played a music and manually tunred on/off the vibration motors.
Thoughts and Next Steps
Compared to the first try, I definitely feel more satisfied. With the increased number of motors, the intensity of the vibrations becomes greater and you can feel the vibrations throughout the area holding the cushion. I feel that I am moving in the right direction.
However, there are still challenges.
The biggest challenge to overcome is the automation of the vibrations to the music. Currently, I am manually vibrating motors to the music, so I can't immerse myself in the experience of vibration. I need to write a python cord so that I can vibrate the motor to match the bass sound.
April 18
3rd Iteration Progress
After some hours of coding, I finally managed to run vibration motors to music. I am so tired now that I cannot update this blog anymore today.
Here is my code. I will update it and add some manuals soon if, by any chance, someone else wants to use it.
https://github.com/bick-jp/audio-motor-sync
April 19
3rd Iteration Progress
I will roughly explain the codes. The purpose of my codes is to power vibration motors to the bass of the music. By referring to Raspberry Pi Spectrum Analyzer, I used the Fast Fourier transform with NumPy.
The codes allow Pi to detect a certain amplitude of the bass sound under 200Hz, and run motors if it is above a certain threshold.
I thought I would have a hard time to match the timing of music and vibrations, but it was easier than I thought it would be. The vibration was always about 0.1 seconds faster than the music, so I adjusted the vibration using Python threading.
I should have used a timestamp to adjust the timing, but I omitted it because of the time and difficulty. It also should be noted that my method above has only been confirmed in my Pi and will not behave the same on all devices.
April 20
3rd Iteration Completed
Here is a video of my 3rd iteretion works. To convey how motors vibrating is hard to tell via the video, but they are vibrating based on the low bass sound under 200Hz.
4th Itereation Idea
The goal is to improve the cushion in terms of wiring strength, aesthetics, and comfortness. I couldn't do much of anything major of those for my 3rd iteration because the cushion is being used for my family's regular life. I am ordering a cushion just for this project and will get it within few days. In the meantime, I will also work on wiring, not using jump wires but to solder wires for the sake of strength.
Sketch
My professor strongly advised me to draw a sketch, so I did.
April 24
4th Iteration
For the 4th iteration, I used a round cushion for better cuddling and also lengthened the wires to make them easier to handle for testing. In my personal opinion, I find a round cushion to be more comfortable to cuddle than a square cushion. It would be more interesting if I could try different shapes and sizes.
Got a round cushion.
Made longer wired version of vibration motors.
Put a cover, and done.
April 25
5th Iteration
In 5th iteration, I have improved the python code. Up until the 4th iteration, you could only play songs that you put in your Raspberry Pi, but with the 5th iteration, you can now play any song by using an external USB device to connect a playback device such as an iphone (like an aux-jack).
As for the programming, the 5th iteration using the sound from the mic jack was a bit difficult. In the beginning I tried PyAudio, but I couldn't synchronize the sound and vibration as I wanted due to delays and errors (input overflow). Ultimately, I solved the problem by using AlsaAudio instead of PyAudio.
The code has been published here. The details will be explained in a clean documentation later. https://github.com/bick-jp/audio-motor-sync
April 26
6th Iteration
In the 6th iteration, I added two push buttons to increase or decrease the value of threshold. The device detects the magnitude of the bass sound at a certain frequency (0-200hz) of a song, and if it's above that threshold, the motors will run. However, the magnitude is different depending on the song, so I added a button so that you can change the threshold manually. It's not aesthetically pleasing, but it's functional and meets my needs for now, so I'll let it go.
April 27
7th Iteration
Due to the schedule of the class, the prototyping will be temporarily completed at the 7th iteration. I'll continue to use and report on this device in person for the next week, but it's good that I've completed a minimum of functions and aesthetics.
For the 7th iteration, the aim was to improve the beauty and comfort of the device by making a cushion cover to fit the round cushion newly purchased. It was very difficult because I had almost no experience in sewing with a sewing machine, but after several hours of difficulty, I managed to get it into shape.
There are a lot of things to improve on, but I'm basically happy with the results. The cushion cover is just the right size and the cushion is just the right amount of firmness without being too hard or too soft. For the material, I used a leftover t-shirt, which is comfortable to the touch and stretches just right. Also, I made a hole for the cable to pass through, so it's easier to handle.
April 28 - May 4
Live With a Robot of Your Own Making
The final assignment for this project is to spend the last week of the semester with a robot I've built and investigate it.
This project began with a very personal motivation in the first place. I was inspired in a situation where I was unable to go to a live concert due to the impact of the COVID-19. The various concerts I was looking forward to, Pearl Jam, Rage Against The Machine, Ozzy Osbourne, Aerosmith, etc, were cancelled. What's more, I couldn't even go to a music studio near my house to play the drums for my hobby. To be honest, I hadn't thought deeply about my project, except that I wanted to put that kind of feeling into it.
What I've found out about myself after living with my robot I've built for the past week is that I'm not concentrating as much as I'd like to. I've been trying out the robot in a variety of ways over the past week. I tried holding it as originally envisioned, sitting on it, lying on it, using it as the back of a chair, while watching something, reading, writing, studying, etc. What I discovered from that experience is that the feeling of satisfaction is completely different when I am focused on the music and vibration than when I am not. There's always something good about the experience when I focus on the music and vibration while using the device. Musically, it is possible to discover new sounds and rhythms, and meditatively, it is possible to reflect on myself and sharpen my concentration. On the contorary, the only thing I've discovered with using this device while doing other work, like writing a class report for example, is that there's nothing good in the experience.
These experiences imply the effectiveness of this device as a meditation when you truly focus on vibration and music. In the same way that you focus on your breathing in zen meditation, you focus on music and vibration when using this device.
Considering my design, what I would like to improve is to strengthen the aspect of meditation, that is, how to design an experiment for a user to FOCUS on the music and the vibration. What I'm thinking about now is adding the hassle of using the device. For example, I could design an environment in which the user can focus on using the device by making it so that only one song can be played, or by making it so that the device won't work unless the user grab a specific location in a specific way, or by making it so that the user can't do anything else while using the device.
The following is a list of vague ideas I have in mind, as a side note.
-
When I go to a concert, does that mean I'm focused? In addition to listening, there are other elements to the concert, such as seeing, experiencing, and interaction of the audience and artists. How do these diverse elements affect concentration and experience?
-
Spotify reported that COVID-19 disrupts ther users listening patterns (Spotify Gains 6 Million Paid Subscribers in Q1 as COVID-19 Disrupts Listening Patterns). It's very interesting how the loss of commuting has eliminated the lifestyle of listening to music on the commute, and how the genre of music that is often heard has changed. I have learned from my experience that listening to music and feeling the vibrations in a focused way is meditative. But it should be thought of as a completely different experience from the casual experience of listening to music.