midi in flash: a proof of concept

Deprecated, please use Chris Wilson’s WebMIDIAPI Shim

I have made a flash movie that is able to receive midi data from a midi device that is connected to your computer. This is accomplished by using a Java applet. The Java applet is receiving the midi data and passes it on to JavaScript and JavaScript sends it to flash via the ExternalInterface.

You can find a proof of concept overhere. You will need to have flashplayer 10 installed because i use the dynamic sound generation possibilities of player 10 for producing sounds when midi note-on data is received.

If you don’t want the hassle of installing flashplayer 10 you can find a flashplayer 9 version overhere. You won’t be able to enjoy the beautiful warm sinus tone generated by flash but you can see flash responding to your midi keyboard. The Release version of Flash Player 10 is now available from the Flash Player Download Center. However, the flashplayer 9 version is still online.

connecting flash to a midi device

When the page has fully loaded, flash asks the Java-applet to investigate your midi setup and displays the result in the yellow part on the top right corner. If a midi in port is detected and available, it will automatically be connected to the flash interface. If not, you can click on one of the other detected devices: you need to look for a midi in port. If Java was not able to detect the kind of midi device (midi in, midi out, synth, sequencer), it will be displayed as ‘device’. A red bullet indicates a successful connection.

So now if you press a key or move a slider on your midi keyboard you should see this represented in the little pink square on the lower right corner. If you play 3 or more notes simultaneously, the purple square on the upper left corner will try to find out what cord you are playing (see below).

<

p style=”text-align: justify;”>
the keyboard and velocity sensitivity

At the center bottom, the matching piano keys will light up on the software keyboard while playing your hardware keyboard. You can also play on this keyboard by clicking the keys with your mouse, but the software keyboard isn’t (yet) velocity sensitive, as opposed to your hardware keyboard. That is: if you have a velocity sensitive keyboard connected.

Because velocity sensitivity of hardware keyboards is supported, i divide the volume of the sound channel by 10. Otherwise the sound might be distorted when you play 2 notes at maximal velocity. By dividing it by 10 this could still happen if you play more than 10 notes at maximum velocity but i assume that you have 10 fingers and won’t use you forearm to play clusters 😉

I have to find another solution though once sustain pedal support is implemented.

pitch bend

In the lower left corner you can set the amount of bend that is used by your pitch bend wheel. You can either pick an interval with the drop down box (from a minor 2nd up to an octave), or choose any value in between using the slider.

the ‘color organ’

I have also implemented a very rudimentary form of a color organ which you’ll find in the black square above the keyboard. You can assign midi controllers or notes to the RGB channels represented by the 3 little accordingly colored squares.

It works like a regular midi-learn/midi-forget toggle that you can find in many offline midi programs: click on the software controller, move the hardware controller, press ‘ok’ and you’re done.

If you connect a note, the velocity is used to change the value of the channel and all notes with the same name in other octaves will be connected as well. This can be interesting: i have experimented a little with assigning specific tonic, subdominant and dominant notes to the channels, like for instance in key of C the note C to the green channel, the note B to the red channel and the note A to the blue channel.

Now when you play you’ll see that colors are more or less matched to harmonic functions: dominant chords will always be a little reddish. The color of a Dmin7 will be, dependent on how hard you play the C and the A, beautifully aquamarine. Cmaj7 is yellow, but if you play the seventh very loud, it will turn more orange.

I have to think up more refined and intelligent ways of connecting tones to harmonies, but i wanted to put this simple color organ in my proof of concept to show as an interesting possibility of a flash midi connection.

the chord finder

This is in fact the reason why started this project: i wanted to create a decent online chord finder. The chords are related to scales and are presented as modi of all scales they fit in. Not all scales are implemented yet, so there are also missing chords.

Only major, minor, minor harmonic and -melodic scales are implemented. Also, only the most common keys are used: C, F, Bb, Eb, Ab, Db, Gb, B, E, A, D, and G. This implies that if you play Db-F-Ab the chord finder will say Db major, but if you lower the third to Fb the chordfinder will enharmonise it to C#-E-G# and thus tell you it is C# minor. This is perfectly sound because C# minor (E major) is far more common then the minor parallel key of Fb (!).

Also this part needs some good reworking because i am not completely happy with the order in which the modi are displayed.

more to come

I am still working on this.

40 Responses to “midi in flash: a proof of concept”


  • Pretty damn cool! I seem to remember there has been research done into the colors of sound that found certain colors to fit better for certain chords. Perhaps you could use this as guideline, if you can find the research of course…

    Would love to see a hookup to roomware server that would make it possible to hook midi input to different types of visualizers…

  • Very interesting, have you done any more work on this?

    Will you make the midi java applet available and let us know how to interact with it? I would love to try out some midi->flash stuff with my electronic drumkit.

  • I am very curious about your experiments! Will put the code on Google Code this weekend.

  • I’ve done some testing on my mac at home, and it seems that it doesn’t work in Firefox 3.0.3/Flash 10. In Safari the midi conf comes up with Synth 1: Java Sound synteziser (which does not work), and Sequencer 1: Real time sequencer (which can be selected, but nothing happens).

    It does not find any of my 3 midi input devices. It would be nice to test some of this out, I hope you get the time to put your code up for some experimentation. As I mentioned I have this idea to use my electronic drumkit to trigger samples and visualizations, but I need help with the midi->flash part.

  • Please try the fp9 version: http://www.abumarkub.net/midi/fp9/
    Since fp10 is not in beta anymore the code needs some adaptions, i am working on it. You can find the Java code of the bridge on Google Code: http://code.google.com/p/miditoflash/

    I am about to upload the as3 code as well. Stay tuned 😉

  • Thanks for the upload, I’ve just had a brief look at it. But I still can’t get any of your examples to work on my setup. I’ve tried the above fp9 version as well, on both my Mac (Safari/Firefox) and through boot camp/Vista (firefox/IE).

    I get a “Detected midi devices on your System”, but nothing happens when I use any of my midi inputs. Hope to have a more detailed look later.

  • Did the list (the yellow one) display any devices? I have changed the order of initializing and tested it on all browsers. Can you check again on your system. Please clear your cache thoroughly first!

    The fp10 version is up and running again: the Event.SAMPLE_DATA appeares to be null in version 10.0.12.36 so i had to set it manually to “sampleData”.

  • In Safari the MIDI CONF list displayed:

    “Synth 1: Java Sound syntheziser
    Sequencer 1: Real time sequencer”

    Only the last one can be selected, but I’m not sure what it is. It is not one of my MIDI inputs.

    In Firefox it displayed:

    “Detected Midi Devices on your system.”

    There is nothing to select.

    I’ve cleared caches and today it does not seem to load the flash (http://www.abumarkub.net/midi/fp10/) at all. It just says Please upgrade your flash player, even though I’m running 10.0.12.36. This happens in both Safari and Firefox on mac, and Firefox & IE on PC. Maybe there is something wrong with the script that embeds the flash? The fp9 version loads but has the same above problems.

  • Correction: The fp10 version loads in Safari after a brief time where the “Please upgrade your flash player” is displayed (like 1 second). It still does not see any of my MIDI inputs. Is there a limit in how many devices it can see?

  • I am afraid something is wrong with your midi setup; i have made a test applet to check your setup. You can find it here:

    http://abumarkub.net/midi/applet/

    The applet does nothing but detecting your midi configuration. Please let me know what the applet displays on your computer.

  • Here is what the applet displays in Safari:

    INPUTS:

    OUTPUTS:

    SYNTHS:
    – Java Sound Synthesizer

    SEQUENCERS:
    – Real Time Sequencer

    OTHER PORTS:

    —————————

    I have two MIDI inputs connected to my mac; One Toneport UX2 and one USB to MIDI Adapter. They work in all programs on my mac, and are visible and active in Audio MIDI Setup in OSX. Maybe there is some bug related to how Java on OSX discoveres MIDI devices?
    The app does not work in Firefox, it just displays “Please wait…”
    What is your test setup like?

  • It looks like the bridge is not working on your system. Try to fix it with mmj: http://www.humatic.de/htools/mmj.htm

    You might want to read this:

    http://createdigitalmotion.com/2007/04/26/why-is-apples-support-for-java-multimedia-so-poor/

  • I tried to use the mmj fix, but it didn’t seem to work on my system. Seems there might be some problems with it. I also tried another, similar mac java midi fix called mandolene MIDI SPI. I get this error in my browser when I try to load your test applet:

    exception: java.security.AccesControlException: access denied (java.util.PropertyPermission user.home read).

    I tried an AIR program called VISP that has its own MIDI server. It works with Mandolene and my drumkit (too bad Mandolene is payware). Anyway, that is good news – MIDI and flash/AIR is possible on a mac.

    http://code.google.com/p/visp-vj/

    I think the source is available here, maybe there are some differences in how you discover/attach MIDI inputs in java?

  • He,

    I have try http://www.abumarkub.net/midi/fp10/.

    but they will stay always on, should be that when you release the key of the keyboard go off.

    Congratulations on your idea, I hope that as soon as we have more news. Thank you

  • That’s weird, what is your setup? Do you see any note-off events in the ‘last midi event’ part? That is the little pink rounded rectangle at the bottom right corner.

  • @Mattis:

    Visp is using OSC which is a totally different approach towards using midi. Allthough OSC has a lot of advantages over midi, i have choosen for an applet beacuse it saves the enduser the hassle of starting a Java service.

    However i have to overcome the current problems on OSX 10.4.8+ and i will be working on that asap. I’ll let you know when it’s done.

  • I’m using windows xp and firefox,

    The note status is always on “Note on””

    it never become “Note Off”

    here is a screenshot http://gerrix.net/midi.jpg

  • And do the other values change while you play? Does your keyboard have a pitch bend, and if so, does it work with the app?

    What is you java version? You can check this as follows:
    – start -> run -> cmd
    – type ‘java -version’
    – hit enter

    It should display something like:

    java version “1.6.0_07”
    Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
    Java HotSpot(TM) Client VM (build 10.0-b23, mixed mode, sharing)

  • Yes, the other values is ok. The colour change when move up, down cursor volume (ctrl 7)

    java -version
    java version “1.6.0_11”
    Java(TM) SE Runtime Environment (build 1.6.0_11-b03)
    Java HotSpot(TM) Client VM (build 11.0-b16, mixed mode, sharing)

  • Hmm, i have to look into this. I will make a test applet so we can test the app step by step. Will take some time though. Sorry for the inconvenience.

    One more thing: what keyboard do you use?

  • Umx49 Behringer (Usb) and Roland piano rd 150 (midi).

    Ok.
    Thanks again =D

  • Hello, how your work is going there?

  • Hi Gerrix, all the best for 2009!

    Unfortunately i haven’t found time yet to fix the Vista bug: i am too busy with commercial projects at this moment. I strive to commit an update that will solve the Vista- as well as the OSX bug before the end of January.

  • this is awesome work, the cleanest simplest solution ive seen. thank you soo much for your work and time invested. I’m new to the flash midi experimentation and would love to be in contact with someone who has this working on their home system. I would also appreciate a little guidance as to how i get it set up here at home. I’m new to the google code system and am not sure wich files i need to have in what directories to make the proof of concept work.
    again hugely impressed with this project. thanks for putting it out there. i’ll get it working someday.

  • Nice write up…usually I never reply to these thing but this time I will,Thanks for the great info.

  • This is really cool. Well done! I’m managing to have created my own application which can accept MIDI input fine, but I’m not too sure how I can send MIDI output to the hardware (Behringer DDX3216 Digital Mixing Desk). Can someone point me in the right direction?

  • Wow! Cool stuff!
    Congratulations!

    I was looking for a flash MIDI synthetizer and found your page. I am really intrested in getting in touch. I love math and music, and I am a pianist and composer, apart from a developer and engineer. I see you have a deep understanding of harmony too. Maybe we could code together.

    Send me an email. It would be intresting to keep in touch.
    Thanks!
    Rod

  • CAn you give me the FLA file of this example .. ?

    http://abumarkub.net/midi/sample/

  • Dear Daniel,

    I’m working on a Art project involving many welknown avantguarde and electronic musicians around the world + University Riverside in California, Technical University Berlin etc. .

    As it looks right now – your work could be very helpfull in realising some of our ideas.

    I’ve already tried the bridge and I’m realy impressed.

    Could you please send me the .fla file. The .fla is not included in the rar package and it would spare some work if I would have the file already.

    Of course I’ll keep you informed abou the done work – and more if we use tghe bridge in the project.

    Thanks a lot – Great Work!!!

    Till soon and all the Best,

    Henry

    Would it be possible to

  • Hi, can I use part of the info found in this post if I put a link back to your blog?

  • hallo can you give the .fla file???
    thanks before dude

  • I am so amazed by your work. My name is Stephen and I am currently a graphic design student at FIT. I’m taking a flash class and for a project I’ve been trying to find a way to make actionscript register keystrokes from a midi keyboard. It looks like you’ve mastered how to do it but your work is a tad advanced for me to follow. Mind helping me out? Any help would be really really appreciated!

Leave a Reply