Aphasia Development Tutorial
From Openmoko
(→This is the sound process so far) |
(→The thing I call a service) |
||
(92 intermediate revisions by one user not shown) | |||
Line 1: | Line 1: | ||
=Installation= | =Installation= | ||
− | + | Your choice: GTA01 use [http://downloads.freesmartphone.org/fso-stable/milestone5.5/ FSO 5.5]. GTA02 use [http://downloads.freesmartphone.org/fso-stable/milestone5.5/ FSO 5.5] or [http://www.openwrt.org OpenWrt]. | |
==OpenWrt== | ==OpenWrt== | ||
− | + | A very configurable Linux distribution, and light on system resources. There's a great forum at [https://forum.openwrt.org/index.php] | |
Go to [[OpenWrt]], section '''Build custom image''' and follow the instructions. When you are at the point '''Configure target and packages''' you have to do some choises. Here is the minimum installation for Aphasia: | Go to [[OpenWrt]], section '''Build custom image''' and follow the instructions. When you are at the point '''Configure target and packages''' you have to do some choises. Here is the minimum installation for Aphasia: | ||
Line 33: | Line 33: | ||
Sound ---> <*> lame | Sound ---> <*> lame | ||
Sound ---> <*> madplay | Sound ---> <*> madplay | ||
+ | Sound ---> <*> flite | ||
+ | Sound ---> <*> sox | ||
Languages ---> <*> erlang | Languages ---> <*> erlang | ||
</pre> | </pre> | ||
+ | |||
+ | For some reason the file libmp3lame.so.0 is missing after installation. You can copy ./build_dir/target-arm_uClibc-0.9.30.1/lame-398-2/ipkg-install/usr/lib/libmp3lame.so.0.0.0 to your neo and then make a softlink to /usr/lib/libmp3lame.so.0 | ||
=== Configure gpsd and ser2net === | === Configure gpsd and ser2net === | ||
Line 69: | Line 73: | ||
=== Install the GUI package ex11 === | === Install the GUI package ex11 === | ||
Take a look at this example of what you can do with ex11 [http://projects.openmoko.org/docman/view.php/59/22/MAQ01113.MP4] | Take a look at this example of what you can do with ex11 [http://projects.openmoko.org/docman/view.php/59/22/MAQ01113.MP4] | ||
− | Take the file at http:// | + | Take the file at [http://www.sics.se/~joe/ex11/] and unzip it in your source directory. |
<pre> | <pre> | ||
mkdir /root/trunk | mkdir /root/trunk | ||
Line 122: | Line 126: | ||
== IDE == | == IDE == | ||
* '''Linux''' I am using SciTE for Erlang development on my Linux box. There are other alternatives but SciTE has some nice features not found in other IDE's. | * '''Linux''' I am using SciTE for Erlang development on my Linux box. There are other alternatives but SciTE has some nice features not found in other IDE's. | ||
− | * '''Windows''' | + | * '''Windows''' SciTE |
− | * '''Mac''' | + | * '''Mac''' SciTE or TextMate. SciTE works on any os x including Snow Leopard. Install MacPorts and mac os x X11 server. Then just "port install SciTE". |
− | === Seting up SciTE for openmoko development === | + | === Seting up SciTE on Linux for openmoko development === |
Before you start configuring SciTE, make sure ssh, scp and rsh works from a terminal window. Read [[USB_Networking]] to get it working. I have to run this script whenever I connect the neo to the host | Before you start configuring SciTE, make sure ssh, scp and rsh works from a terminal window. Read [[USB_Networking]] to get it working. I have to run this script whenever I connect the neo to the host | ||
<pre> | <pre> | ||
Line 179: | Line 183: | ||
= Architecture = | = Architecture = | ||
− | Finally | + | Finally it's starting to be interesting, how is the Aphasia system built? It's not very common to build systems in [http://www.pragprog.com/articles/erlang Erlang], and definitely rare to build GUI code in it. If you want to check out how fast a GUI can be on a soft real-time system like erlang, take a look at this video [http://www.kvamme.se/Aphasia/Images/erlang_neo2.mpg clip]. Listen to the finger tapping the screen and notice there is almost no latency at all. |
== Graphics == | == Graphics == | ||
− | I'm using vector graphics to produce new widgets. Vector graphics is lightweight and programmable, easy to scale and rotate, and the bytesize is small. Use any vector-capable drawing application. I | + | I'm using vector graphics to produce new widgets. Vector graphics is lightweight and programmable, easy to scale and rotate, and the bytesize is small. Use any vector-capable drawing application. I use an opensource 2D drawing software, [http://www.ribbonsoft.com/qcad.html QCAD] to create the GUI components. Save as DXF file format and use dxf2erl to convert the cad drawing to erlang source code. Take dxf2erl.erl from [https://sourceforge.net/projects/medicalmobility/files/ here] |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
== Where are all the menus? == | == Where are all the menus? == | ||
Line 198: | Line 194: | ||
== The thing I call a service == | == The thing I call a service == | ||
− | A service is often described as a contract between a consumer and a producer. When you go to the barber you know you will get a shave (if you are a man). The service is the implicit contract between you and the barber that he will shave not rub. There are many barber shops down the street offering essentially the same service, so there can be many service producers for one service consumer. | + | A service is often described as a contract between a consumer and a producer. When you go to the barber you know you will get a shave (if you are a man). The service is the implicit contract between you and the barber that he will shave, not rub. |
+ | |||
+ | There are many barber shops down the street offering essentially the same service, so there can be many service producers for one service consumer. | ||
− | And in the Aphasia system, each service producer is a separate Erlang process. All services are running in parallell, just like in real life. The Erlang VM is quite different from other platforms | + | And in the Aphasia system, each service producer is a separate Erlang process. All services are running in parallell, just like in real life. The Erlang VM is quite different from other platforms, thousands or tens of thousands of processes running at the same time is not uncommon in a well designed Erlang system. |
== The GSM Service producer == | == The GSM Service producer == | ||
Line 218: | Line 216: | ||
== The GPS process == | == The GPS process == | ||
Just like GSM, this process is invisible and always on. It can on enquiry reply current Lat/Long. It can also periodically send current position to another process. It is using GPSD as it's source of information. As you probably notice by now, each Erlang process is very small and is compliant to the UNIX pattern "do one thing and do it well". The GPS process is currently only 50 lines of code. | Just like GSM, this process is invisible and always on. It can on enquiry reply current Lat/Long. It can also periodically send current position to another process. It is using GPSD as it's source of information. As you probably notice by now, each Erlang process is very small and is compliant to the UNIX pattern "do one thing and do it well". The GPS process is currently only 50 lines of code. | ||
+ | |||
+ | The GPS process is constantly recording a track point file in standard Google Maps GPX format. [http://www.kvamme.se/Aphasia/mokotrack63419284517.gpx Here] is a sample recorded at the 9 km running track in a recreational area in Gothenburg. [http://www.kvamme.se/Aphasia/9km.kmz Here] is the same file converted to a Google Earth track by [http://www.gpsvisualizer.com GPS Visualizer]. | ||
== The Sound process == | == The Sound process == | ||
− | Started to code the Sound process (Sound Service Provider - remember ;) this weekend. It's using madplay and aplay together to be able to play mp3's: '''madplay --output=wave:- /root/" ++ Tune ++ " | aplay --buffer-size=20480'''. The service can control volume with a call to '''amixer | + | Started to code the Sound process (Sound Service Provider - remember ;) this weekend. It's using madplay and aplay together to be able to play mp3's: '''madplay --output=wave:- /root/" ++ Tune ++ " | aplay --buffer-size=20480'''. The service can control volume with a call to '''amixer sset "PCM" VolumeValue'''. |
=== This is the sound process so far === | === This is the sound process so far === | ||
− | + | (if you want the latest version, look in the [http://github.com/skvamme/MedicalMobility project] area) | |
<pre> | <pre> | ||
-module(sound). | -module(sound). | ||
Line 272: | Line 272: | ||
'''-vsn("$Id").''' This is a tag for my configuration management system, subversion. You don't need versioning but it's nice to know you can back out of a change in code that just didn't work. | '''-vsn("$Id").''' This is a tag for my configuration management system, subversion. You don't need versioning but it's nice to know you can back out of a change in code that just didn't work. | ||
− | '''-export([make/1]).''' This one is important. It tells which functions are public. make() can be called from another module just by a call to sound:make(xxx). | + | '''-export([make/1]).''' This one is important. It tells which functions are public. make() can be called from another module just by a call to sound:make(xxx). The /1 indicates it takes only one argument. If you want all your functions to be public you can use '''-compile'''(export_all). |
+ | |||
+ | '''make(Parent) -> spawn_link(fun() -> init(Parent) end).''' make(Parent) is declared as public and all it does is to spawn (create) a new process and call the init function in that process. Creating a process in erlang is cheap and is comparable to creating a new object in eg. Java. | ||
+ | |||
+ | '''init(Parent) ->''' | ||
+ | '''io:format("Sound process is started~n",[]),''' | ||
+ | '''loop(Parent,null).''' This process is called by make() as the first thing to do after creating the new process. The name is not important, it can be anything, but it does one imprtant thing except from printing a log message. It is starting the loop. | ||
+ | |||
+ | '''loop(Parent,Port) ->''' The loop! Every erlang process have it's own private mailbox. This loop is the place where all incoming messages are taken care of. | ||
+ | |||
+ | '''receive''' opens the mailbox and acts just like a case statement. | ||
+ | |||
+ | '''{setsound,cpugsmhandset} ->''' If the recieved message looks exactly like this. the following lines of code are executed. | ||
+ | |||
+ | '''os:cmd("amixer -d sset \"PCM\" 255"),''' This code calls the module os and a function in that module, cmd(). Yes, it's executing an os command, amixer with a few parameters. | ||
+ | |||
+ | '''end.''' Ends the receive command. | ||
+ | |||
+ | = [http://www.ibm.com/developerworks/webservices/library/ar-soamat/index.html Higher Order Services] = | ||
+ | |||
+ | Up till now we have created services serving other services. Now it's about time to show a top level service, one that is serving the user. One that can be seen and touched. | ||
+ | <gallery> | ||
+ | Image:Sound2.jpeg|''output service'' | ||
+ | Image:Dialer2.jpeg|''dialer service'' | ||
+ | Image:Clock2.jpeg|''clock'' | ||
+ | Image:Mimic2.jpeg|''mimic service'' | ||
+ | Each service has a location. Moving around is done by swiping in different directions, movie clip below.|[http://www.kvamme.se/Aphasia/Images/erlang_neo.mpg] | ||
+ | Another, cooler way to change service is to take a walk down the street. The GPS service is doing the swipin' for you, and the services for a certain location are showing up on screen when you move around. | ||
+ | When you walk up to the bus stop the GPS service makes sure you have the clock and the bus_traveling service at hand. What the bus_traveling service should do is up to you and your doktor. | ||
+ | </gallery> | ||
+ | |||
+ | == Output == | ||
+ | |||
+ | === The handset process === | ||
+ | === The speaker process === | ||
+ | === The headset process === | ||
+ | === The volume process === | ||
+ | |||
+ | == Dialer == | ||
+ | |||
+ | === Each figure is a process === | ||
+ | |||
+ | == Clock == | ||
+ | === The clock process === | ||
+ | == Mimic == | ||
+ | === The mp3 process === | ||
+ | === The mimic process === | ||
[[Category:Application Developer]] | [[Category:Application Developer]] |
Latest revision as of 23:35, 3 February 2010
Contents |
[edit] Installation
Your choice: GTA01 use FSO 5.5. GTA02 use FSO 5.5 or OpenWrt.
[edit] OpenWrt
A very configurable Linux distribution, and light on system resources. There's a great forum at [2]
Go to OpenWrt, section Build custom image and follow the instructions. When you are at the point Configure target and packages you have to do some choises. Here is the minimum installation for Aphasia:
make menuconf OpenWrt Kamikaze (r16871) Configuration Target System (Samsung S3C24xx [2.6]) ---> Samsung S3C24xx [2.6] Target Profile (Openmoko GTA-02 (minimal)) ---> Openmoko GTA-02 (minimal) [*] Image configuration ---> (192.168.0.202) LAN IP Address Xorg ---> app ---> <*> xauth Xorg ---> driver ---> <*> xf86-input-tslib Xorg ---> driver ---> <*> xf86-video-glamo Xorg ---> font ---> <*> dejavu-fonts-ttf Network ---> <*> gpsd Network ---> <*> ser2net Phone ---> <*> freerunner-alsa-scenarios Utilities ---> <*> alsa-utils Utilities ---> <*> bash Extra packages ---> <*> zoneinfo-europe Sound ---> <*> lame Sound ---> <*> madplay Sound ---> <*> flite Sound ---> <*> sox Languages ---> <*> erlang
For some reason the file libmp3lame.so.0 is missing after installation. You can copy ./build_dir/target-arm_uClibc-0.9.30.1/lame-398-2/ipkg-install/usr/lib/libmp3lame.so.0.0.0 to your neo and then make a softlink to /usr/lib/libmp3lame.so.0
[edit] Configure gpsd and ser2net
To autostart ser2net and gpsd insert a line in /etc/init.d/httpd, last line in desktop section:
desktop() { config_foreach system_config system hostname="${hostname:-OpenWrt}" unset args config_load httpd [ "$?" != "0" ] && { uci_set_default httpd <<EOF config 'httpd' option 'port' '80' option 'home' '/www' EOF config_load httpd } config_foreach httpd_config httpd /usr/sbin/ser2net ; /usr/sbin/gpsd /dev/ttySAC1 }
Open /etc/ser2net.conf and replace all port examples at the very end with this single line:
2001:raw:0:/dev/ttySAC0:115200 NONE 1STOPBIT 8DATABITS -XONXOFF -LOCAL RTSCTS
[edit] Runtime Environment
[edit] Install the GUI package ex11
Take a look at this example of what you can do with ex11 [3] Take the file at [4] and unzip it in your source directory.
mkdir /root/trunk cd /root/trunk tar -xvf ex11_lib.tgz
In file /root/.profile
export DISPLAY=:0
In file /etc/hosts remove . (dot) at the end of line localhost.
127.0.0.1 localhost
In file /root/.erlang
io:format("Running Erlang from root~n"). code:add_patha("/root/trunk").
Copy Xauthority from host
scp .Xauthority phone:/root
Add localhost to it with xauth
xauth add OpenWrt/unix:0 MIT-MAGIC-COOKIE-1 cfcc5ef98f9718f90154f355c0ae9f62 exit cp .Xauthority /
Include erlang VM in the boot process, simply add a line to /etc/init.d/x11 right after Xorg:
start() { export DISPLAY=":0" config_load x11 Xorg $ARGS & erl -setcookie SFEWRG34AFDSGAFG35235 -name neo@192.168.0.202 -noshell -pa /root/trunk -s main start > debug.txt }
[edit] Development Environment
This section is about setting up and do some work on your development system. It can be a PC with Linux or Windows, or it can be a Mac.
[edit] IDE
- Linux I am using SciTE for Erlang development on my Linux box. There are other alternatives but SciTE has some nice features not found in other IDE's.
- Windows SciTE
- Mac SciTE or TextMate. SciTE works on any os x including Snow Leopard. Install MacPorts and mac os x X11 server. Then just "port install SciTE".
[edit] Seting up SciTE on Linux for openmoko development
Before you start configuring SciTE, make sure ssh, scp and rsh works from a terminal window. Read USB_Networking to get it working. I have to run this script whenever I connect the neo to the host
#!/bin/bash ifconfig usb0 192.168.0.200 netmask 255.255.255.0 /sbin/route add -host 192.168.0.202/32 dev usb0 echo 1 > /proc/sys/net/ipv4/ip_forward iptables -A POSTROUTING -t nat -j MASQUERADE -s 192.168.0.0/24 iptables -P FORWARD ACCEPT
Download from here http://www.scintilla.org/SciTE.html and install. If you have apt-get it can be as easy as
sudo apt-get install scite
Start SciTE and select menu Options|Open Global Options and uncomment
#Erlang|erl||\
and at other lines where erlang is a comment. You may have to start SciTE as root to do this. (To uncomment you just delete the hash at the beginning of the line).
Create a file named SciTE.properties in your erlang source directory.
# SciTE.properties is the per directory local options file and can be used to # override settings made in SciTEGlobal.properties, SciTEUser.properties and # SciTEDirectory.properties. command.compile.*.erl=erlc $(FileName).erl command.go.subsystem.*.erl=0 command.go.*.erl=erl -noshell -s $(FileName) start command.subsystem.1.*.erl=0 command.name.1.*.erl=svn update command.1.*.erl= svn update $(FileNameExt) command.name.2.*.erl= svn ci command.2.*.erl= svn ci -m '$(4)' $(FileNameExt) command.name.3.*.erl=svn co command.3.*.erl=*svn co $(1) command.name.4.*.erl=svn status command.4.*.erl= svn status command.name.5.*.erl=hot code loading command.5.*.erl=erl -noshell -setcookie SFEWRG34AFDSGAFG35235 -name moko2@192.168.0.200 -eval "net_adm:ping('neo@192.168.0.202'), c:nl($(FileName))." -s init stop command.subsystem.6.*.erl=0 command.name.6.*.erl=copy to Freerunner command.6.*.erl= scp $(FileName).beam phone:/root/trunk/ command.subsystem.7.*.erl=0 command.name.7.*.erl=add to subversion command.7.*.erl= svn add $(FileNameExt) abbreviations.*.erl=$(SciteUserHome)/erl_abbrev.properties line.margin.visible=1 line.margin.width=2+ style.*.33
[edit] Architecture
Finally it's starting to be interesting, how is the Aphasia system built? It's not very common to build systems in Erlang, and definitely rare to build GUI code in it. If you want to check out how fast a GUI can be on a soft real-time system like erlang, take a look at this video clip. Listen to the finger tapping the screen and notice there is almost no latency at all.
[edit] Graphics
I'm using vector graphics to produce new widgets. Vector graphics is lightweight and programmable, easy to scale and rotate, and the bytesize is small. Use any vector-capable drawing application. I use an opensource 2D drawing software, QCAD to create the GUI components. Save as DXF file format and use dxf2erl to convert the cad drawing to erlang source code. Take dxf2erl.erl from here
[edit]
Aphasia is built for people with different communicative disabilities. Menus where ruled out from the Aphasia GUI from the very beginning due to the fact that you don't even have to have a disability to get lost in a miniature menu system.
Instead I'm using a positioning system where services (I'll talk about those later) are laid out on a large surface (surface of the earth actually) and you can reach a service by swiping the screen in different directions. Another much cooler way to reach a service is when you physically go to a certain place in "real life" (like Jane did in this narrative).
[edit] The thing I call a service
A service is often described as a contract between a consumer and a producer. When you go to the barber you know you will get a shave (if you are a man). The service is the implicit contract between you and the barber that he will shave, not rub.
There are many barber shops down the street offering essentially the same service, so there can be many service producers for one service consumer.
And in the Aphasia system, each service producer is a separate Erlang process. All services are running in parallell, just like in real life. The Erlang VM is quite different from other platforms, thousands or tens of thousands of processes running at the same time is not uncommon in a well designed Erlang system.
[edit] The GSM Service producer
Where should we start if not with the GSM service producer (I will use the name GSM process from now on). What kind of service can we expect from the GSM process, can we expect it to offer a bearer between our phone and some other. The kind of payload we are sending on that bearer is not stated in the service.
The GSM process is always on, it's also invisible. Most processes can display themselves on the screen but the GSM process does never show up, it's running in the background listening and reacting to the different signals coming from the Calypso GSM chip via the ser2net deamon. It is also checking it's own mailbox for messages. Every Erlang process have a private mailbox. It can also send messages to other processes.
If the Calypso GSM chip gets an incomming call, the GSM process sends a message to the Userland process to "go to and start" the Dialer process (remember every process have a GPS position). It also sends a message to the Sound process to act on an incoming call. All this takes no time at all, Erlang is soft-realtime, and at the same moment the caller hears the first ring-tone the neo starts to play (my neo plays Gasoline "This is my life").
The Userland process is the mother of all other processes, and can, if something nasty should happen in the Dialer process, kill it off and start a new Dialer process. That's how Erlang can have 5 nines.
[edit] Aphasia Userland Process
The mother of all Aphasia processes. This process can display itself on the screen. It's an empty fullscreen window. And that is the first thing it does, displays itself. Then it creates the backend processes, the Sound process, the GSM process that you already know about, and the GPS process I only mentioned before.
Then it goes into it's main loop. All Erlang processes have a main loop where they receive messages and send messages (if they are built to stay alive).
[edit] The GPS process
Just like GSM, this process is invisible and always on. It can on enquiry reply current Lat/Long. It can also periodically send current position to another process. It is using GPSD as it's source of information. As you probably notice by now, each Erlang process is very small and is compliant to the UNIX pattern "do one thing and do it well". The GPS process is currently only 50 lines of code.
The GPS process is constantly recording a track point file in standard Google Maps GPX format. Here is a sample recorded at the 9 km running track in a recreational area in Gothenburg. Here is the same file converted to a Google Earth track by GPS Visualizer.
[edit] The Sound process
Started to code the Sound process (Sound Service Provider - remember ;) this weekend. It's using madplay and aplay together to be able to play mp3's: madplay --output=wave:- /root/" ++ Tune ++ " | aplay --buffer-size=20480. The service can control volume with a call to amixer sset "PCM" VolumeValue.
[edit] This is the sound process so far
(if you want the latest version, look in the project area)
-module(sound). -author(skvamme). -vsn("$Id"). -export([make/1]). make(Parent) -> spawn_link(fun() -> init(Parent) end). init(Parent) -> io:format("Sound process is started~n",[]), loop(Parent,null). loop(Parent,Port) -> receive {setsound,cpugsmhandset} -> os:cmd("amixer -d sset \"PCM\" 255"), os:cmd("amixer -d sset \"Mono Mixer Left\" on"), os:cmd("amixer -d sset \"Mono Mixer Right\" on"), os:cmd("amixer -d sset \"Left Mixer Left\" on"), os:cmd("amixer -d sset \"Right Mixer Right\" on"), ?MODULE:loop(Parent,Port); {stop} -> os:cmd("killall madplay"), ?MODULE:loop(Parent,Port); {play,Tune} -> D = "madplay --output=wave:- /root/" ++ Tune ++ " | aplay --buffer-size=20480", Port1 = open_port({spawn,D}, [stream, use_stdio, stderr_to_stdout]), ?MODULE:loop(Parent,Port1); {setsound,Type} -> os:cmd("alsactl -f /usr/share/openmoko/scenarios/" ++ Type ++ ".state restore"), ?MODULE:loop(Parent,Port); {volume, up} -> os:cmd("amixer -d sset \"PCM\" 255"), ?MODULE:loop(Parent,Port); Any -> io:format("Sound got unknown msg:~p~n",[Any]), ?MODULE:loop(Parent,Port) end.
As you can see it's quite simple, let's single-step it.
-module(sound). Every piece of erlang code must have a name. An erlang module is nothing but a piece of code. With a name, for your convienience, so you can easily use it.
-vsn("$Id"). This is a tag for my configuration management system, subversion. You don't need versioning but it's nice to know you can back out of a change in code that just didn't work.
-export([make/1]). This one is important. It tells which functions are public. make() can be called from another module just by a call to sound:make(xxx). The /1 indicates it takes only one argument. If you want all your functions to be public you can use -compile(export_all).
make(Parent) -> spawn_link(fun() -> init(Parent) end). make(Parent) is declared as public and all it does is to spawn (create) a new process and call the init function in that process. Creating a process in erlang is cheap and is comparable to creating a new object in eg. Java.
init(Parent) -> io:format("Sound process is started~n",[]), loop(Parent,null). This process is called by make() as the first thing to do after creating the new process. The name is not important, it can be anything, but it does one imprtant thing except from printing a log message. It is starting the loop.
loop(Parent,Port) -> The loop! Every erlang process have it's own private mailbox. This loop is the place where all incoming messages are taken care of.
receive opens the mailbox and acts just like a case statement.
{setsound,cpugsmhandset} -> If the recieved message looks exactly like this. the following lines of code are executed.
os:cmd("amixer -d sset \"PCM\" 255"), This code calls the module os and a function in that module, cmd(). Yes, it's executing an os command, amixer with a few parameters.
end. Ends the receive command.
[edit] Higher Order Services
Up till now we have created services serving other services. Now it's about time to show a top level service, one that is serving the user. One that can be seen and touched.
- Each service has a location. Moving around is done by swiping in different directions, movie clip below.
- Another, cooler way to change service is to take a walk down the street. The GPS service is doing the swipin' for you, and the services for a certain location are showing up on screen when you move around.
- When you walk up to the bus stop the GPS service makes sure you have the clock and the bus traveling service at hand. What the bus traveling service should do is up to you and your doktor.