Android Concurrent Programming: Looper, Handler and Message

Class Handler is used extensively by Android Applications and framework to achieve concurrent behavior. A typical usage is to allow the worker thread to post an action to be run in the UI thread. Similar C++ constructs, with a little variant, are also widely used by Android native subsystem such as SurfaceFlinger and StageFright to implemented sophisticated concurrent graphic operation and media codec pipeline.

In this article, we will examine the main components that enables this concurrent programming model, in a language neutral way. Let's start with the class diagram: 
Thread, Looper, Handler, Message Class Diagram
A. Each Thread has 0 or 1 Looper.  Not every Thread is born with a Looper. If you need one, call Looper.prepare() and then Looper.loop() in the Thread you want the Looper run in, or use HandlerThread.

Activity's main UI thread already has a Looper created by the framework and it can be retrieved by calling Looper.getMainLooper().

B. Each Looper has exactly 1 MessageQueue. The Looper.loop() keeps polling the MessageQueue and process the Message if any.

C. Handler is the primary user facing class and it is where the customized message handling code goes. It is responsible for creating the message (obtainMessage()), sending the message (sendMessage()) and handling the message(handleMessage()).  Each Handler must be attached to a Looper. When calling sendMessage(), the Message will be sent to the MessageQueue owned by the Handler's Looper.  Handler is the Message producer , while the Looper is the Message consumer.

The most important thing to remember is sendMessage() can be called in any threads but the handler code (i.e handleMessage()) will always be running in the Thread which the Handler's Looper resides in.

D. Multiply Handlers can associate to the same Looper, with each Handler dealing with a particular type of messages. In fact, before you creating a customized Handler to talk to the main thread as described previously, there are already several Handlers[1] created by the framework to deal with system events.

[1] ActivityThread::H() , ViewRootImpl::ViewRootHandler

E. Each Message must have a target Handler, who knows how to handle the message and where to send the Message.  sendToTarget() is actually a handy wrapper to mHandler.sendMessage(this)

Following diagram illustrates the steps of writing a customize Handler in the Main UI Thread and the Message sending/handling flow. Hopefully, the diagram is self explanatory and I won't bother to write more text for it :)
Creating A Customized Handler in Android Main UI Thread and The Message Flow
Actually, the Handler and its relate classes can be considered as an Android implementation of concurrent design pattern called Active Object. Understanding the implementation details can make us more confident when using those utilities, or to certain extend, make us a better programmer.

Yet Another Deep Dive of Android Binder Framework, Part 1

There are already lots of excellent articles/presentations on the web talking about Android Binder. Most of them are focusing on the Java Service/AIDL, or talking only about the theory instead of looking at the actually classes involved. In this series of articles, we will take a close look at the major classes that form the Android Binder Framework (user space only), as shown in Figure 1 below. These classes are the foundation for implementing Java service and are the backbones of all system native services. And, of particular interest, we will talk about the implementation of several important binder features such as passing IBinder object or file descriptors across process.  



Figure 1 Android Binder Framework

IBinder, BpBinder and BBinder



IBinder defines the abstract interface for a binder object; BpBinder and BBinder are the concrete implementation in proxy and server side, respectively. Following table lists the most important methods of IBinder interface, and their corresponding implementation in BpBinder and BBinder.


IBinder
BpBinder
BBinder
constructor
BpBinder(int32_t handle)
Nothing special
transact(code,data,reply,flag)
call IPCThreadState::transact(mHandle)
call onTransact() overrided in subclass
linkToDeath
valid
N/A
localBinder(): BBinder *
return NULL
return this
remoteBinder():BpBinder *
return this
return NULL
queryLocalInterface
return NULL
return this


That constructor of BpBinder takes an integer handle, which is created by binder driver. When calling BpBinder::transact(), that handle will be passed to the binder driver and used by the driver to locate the transaction’s target process and target BBinder object. Once the BBinder object is found, binder driver will wake up one of the target process’s binder threads and invoke the BBinder’s transact() method. This describes briefly how the BpBinder and BBinder are connected and we will revisit again in following example of adding a customer interface (or services).


IInterface, BpInterface and BnInterface



The Binder classes we talked about in previous section deal with the low level inter-process communication, on top of the binder driver. The IInterface and it associated classes are to provide higher level abstraction, turning a inter-process communication into a remote procedure call.


To glue all those classes together and see how they interact with each other, we'll take a look at a simple example of implementing a simple interface, IAdd, which provides only single method of adding two int values. 

Here are the four steps needed to set up the binder based client/server architectures. You can see this pattern everywhere in Android native framework.


  1. Define IAdd by subclassing IInterface
  2. Subclass BpInterface with BpAdd, and implement interface method add
    1. Marshall the request data
    2. Call remote()->transact(code,data,reply) to send the transaction to bind driver
    3. Wait and unmarshall the reply
class BpAdd : public BpInterface {};

int  BpAdd::add(int a, int b)
{
       Parcel data, reply;
       // 2.a.
       data.writeInterfaceToken(IAdd::getInterfaceDescriptor());
       data.writeInt32(a);
       data.writeInt32(b);
       // 2.b
       if (remote()->transact(ADD, data, &reply) == NO_ERROR) {
           // 2.c
           int result = reply.readInt32();
           return result;
    }
   // on error, for simplicity, assume -1 means error.
   return -1;
}


  1. Subclass BnInterace with BnAdd, and implement onTransact()
    1. Dispatching based on transaction code
    2. Call the real implementation in the subclass of this class - see step 4
    3. Marshall the result into reply, which will be sent to driver and be received in the BpAdd
class BnAdd : public BnInterface {};

status_t BnAdd::onTransact(
   uint32_t code, const Parcel& data, Parcel* reply, uint32_t flags)
{
   switch(code) {
       // 3.a
       case ADD: {
           CHECK_INTERFACE(IAdd, data, reply);
           int a = data.readInt32();
           int b = data.readInt32();
           // 3.b
           int result = add(a, b);     //implemented in AddImpl
           // 3.c
           reply->writeInt32(result);
           return NO_ERROR;
       } break;
       default:
           return BBinder::onTransact(code, data, reply, flags);
   }
}


4.  Subclass BnAdd with AddImpl, which provide the real implementation.
class AddImpl : public BnAdd {};

int AddImpl::add(int a, int b) {
   return a + b;
}


Figure 2 is what the class hierarchy looks like after adding our customer service; the red and blue line indicate the flow of calling its method remotely from the proxy.
Figure 2 Implement Custom Interface and Method Calling flow
                             
So far, we have covered the basic of the Android user space binder framework: the Binder classes, the Interface classes and how to add, implement and use new interface. You may have noticed that I have ignored some functions such as Parcel.writeStrongBinder and IInterface::asBinder and wondering what they are for. 

Yes, those are the real interesting stuff in Binder. In next article, we will take about how to pass an IBinder object across process, how to convert between the IBinder and IInterface, and when and who create the BpBinder object. We will also revisit in more detail regarding how the binder driver set up the handle <-> object mapping.

Introduction to Android C++ reference counting, part 2

In last article, we introduced the strong pointer, or sp. Today, we will take about another type of smart pointer - weak pointer, or wp as called in Android. Let's look at an example.

using namespace android;

// class Memory subclass RefBase so it can be reference counted
// and be accepted by both sp<T> and wp<T>, where sp stands for
// strong pointer and wp stands for weak pointer. 
// A weak pointer won't prevent the object it points to from being 
// deleted - as long as there is no strong pointer pointing to the 
// raw object it will be deleted. Hence, to make sure the raw object
// is still valid, you will first need to promote it to be strong 
// pointer first. If the promote success, use it; otherwise, you know
// underlying object is no longer valid and you won't want to use it.  

class Memory: public RefBase {
public:
    Memory(int size) : mSize(size), mData(NULL) {
        ALOGD("        Memory constructor %p ",this);
    }

    virtual ~Memory() {
        ALOGD("        Memory destructor %p", this);
        if (mData)  free(mData);
    }

    virtual void onFirstRef() {
        ALOGD("        onFirstRef on %p",this);
        mData = malloc(mSize);
    }

    int size() {return mSize;}
private:
    int mSize;
    void *mData;
};

// used as a MARK in the output
#define L(N)   ALOGD("LINE %d TRIGGER:",N);
// print out the strong counter numbers of the object
#define C(obj) ALOGD("        Count of %p : %d", (void*)obj, obj->getStrongCount());

int main()
{
    wp<Memory> wpm1;
    Memory *m = new Memory(1);
    {//scope for spm1
        // create a Memory instance and let it be managed by a sp
        L(1)
        sp<Memory> spm1(m); 
        // assign the sp to the weak pointer, wpm1
        wpm1 = spm1;

        { // scope for spm2
            // You can't access wp's underlying raw pointer directly, since
            // there is no accessor apis defined for it.
            // Recall that for sp, there are three accessor apis/operators
            // you can use: get(),*,->
            // To access the raw pointer the wp pointing to, you need first
            // promote it to an sp, and use the returning sp if it is not NULL.
            sp<Memory> spm2 = wpm1.promote();
            if (spm2 != NULL) {
                L(2)
                ALOGD("        Promotion to sp successfully");
                spm2->size();
                // ref count is 2, since both spm1 and spm2 pointed to it
                C(m)
            }
        }
        // at this point, spm2 is out of scope, ref count of the m becomes 1 
    }
    // Beyond previous close curly, spm1 was also out of scope.
    // Since there was no strong pointer to m, m was destroyed.
    // Hence, following promotion will fail and we should not try to
    // use it any more.
    L(3)
    sp<Memory> spm3 = wpm1.promote();
    if (spm3 == NULL) {
        ALOGD("        Promotion Fail. Object already been destructed");
    }

    L(-1)
    return 0;
}

And, here is the output.

LINE 1 TRIGGER:
        Memory constructor 0x5597001050 
        onFirstRef on 0x5597001050
LINE 2 TRIGGER:
        Promotion to sp successfully
        Count of 0x5597001050 : 2
        Memory destructor 0x5597001050
LINE 3 TRIGGER:
        Promotion Fail. Object already been destructed
LINE -1 TRIGGER:

You maybe wondering what is the usage of weak pointer.  Here are two examples:
  1. hold a weak reference to observers. Only notify observer when it is alive.
  2. To solve the memory leak issue involved in the circular strong reference. Suppose A hold a strong reference to B and B hold strong reference to A. Then, neither A or B will be destroyed since there is always one strong reference to them. One of the solution is to make one of reference be weak reference.  
To wrap up, you may want to check out c++11 newly introduced std::shared_ptr and std::weak_ptr. The first one maps to sp and the later wp.

Introduction to Android C++ reference counting, part 1

Anyone working on the Android native framework can't get around several utility classes that are used almost everywhere by the native C++ code. sp, or StrongPointer, is one of them. It is vital important to understand how it works so that you can understand the code precisely and write code that is clean and won't leak resource. In this article, we will cover the basic ideas and usage of sp, by examples.

#include <utils/RefBase.h>
#include <utils/Log.h>
#include <cstdlib>
#include <cassert>

using namespace android;

// class Memory subclass RefBase[1] so it can be reference counted
// and be accepted by template class sp<T> [2], where the sp stands for
// strong pointer
// [1]https://android.googlesource.com/platform/frameworks/native/+/jb-mr1-dev/include/utils/RefBase.h
// [2]https://android.googlesource.com/platform/frameworks/native/+/jb-mr1-dev/include/utils/StrongPointer.h

class Memory: public RefBase {
public:
    Memory(int size) : mSize(size), mData(NULL) {
        ALOGD("        Memory constructor %p ",this);
    }

    virtual ~Memory() {
        ALOGD("        Memory destructor %p", this);
        if (mData)  free(mData);
    }

    virtual void onFirstRef() {
        ALOGD("        onFirstRef on %p",this);
        mData = malloc(mSize);
    }

    int size() {return mSize;}
private:
    int mSize;
    void *mData;
};

// used as a marker in the output log
#define L(N)   ALOGD("LINE %d TRIGGER:",N);
// print out the strong counter number of the object
#define C(obj) ALOGD("        Count of %p : %d", (void*)obj, obj->getStrongCount());

int main()
{
    {
        // create a Memory instance and assign it to a raw pointer
        L(1)
        Memory *m1 = new Memory(4);
        // create a strong pointer, using constructor sp(T* other),
        // which will increase m1's reference counter by 1, and
        // call m1::onFirstRef, where you can do the lazy initialzation
        L(2)
        sp<Memory> spm1 = m1;
        C(m1);

        // usually, we will combine previous two steps into one single statement.
        // create another strong pointer, spm2, and initialize it.
        // To get the raw object, use sp<T>::get()
        L(3)
        sp<Memory> spm2 = new Memory(128);
        Memory *m2 = spm2.get();
        // to access the method, use sp as if you are working with raw pointer
        int size = spm2->size();

        // create a 3rd sp,spm3,using constructor sp(const sp<T>& other),
        // which will increase the reference counter pointed by spm1 by 1.
        // now, m1 is pointed by two strong pointers, spm1 and spm3
        L(4)
        sp<Memory> spm3 = spm1;
        C(m1);

        // below are same as L(4), except that the scope of spm4 is within the block
        L(5)
        {
            sp<Memory> spm4 = spm1;
            C(m1);
            // at this point, m1 is pointed by spm1, spm3 and spm4
        }

        // beyond this point, spm4 is destructed and no longer point to m1
        // so, the reference of m1 is still 2, i,e pointed by spm1, spm3
        L(6)
        C(m1);

        // trigger sp& operator = (const sp<T>& other);
        L(7)
        // before the assigment, spm2 pointed to m2 and spm3 pointed to m1
        spm3 = spm2;
        // after the assigment, spm3 will no longer point to m1 but m2.
        // so the reference counter of m1 decrease by 1 and the reference counter
        // of m2 increase by 1.
        // now , m1 is pointed by spm1 m2 is pointed by spm2 and spm3
        C(m1);
        C(m2);

        // spm5 is a reference to spm1, no new strong object is created.
        // so reference count of m1 stays the same
        L(8)
        sp<Memory> &spm5 = spm1;
        C(m1);

        // we can also create a smart pointer pointing to nothing at first
        // and later assign it a value. We can also remove the reference explictly
        // by calling sp::clear()
        L(9)
        sp<Memory> spm6;
        assert(spm6.get() == NULL);
        spm6 = spm1;
        C(m1);
        L(10)
        spm6.clear();
        assert(spm6.get() == NULL);
        C(m1);

    }
    // beyond the close curly, all the smart pointer objects are out of
    // scope, so they will be desctructed and cause the reference counter
    // of its managed object decrease by 1. For example,
    // when both spm1 and spm6 destructs,the reference count of m1 decrease to 0
    // and it will trigger the destructor of m1.
    L(-1)
    return 0;
}

And, below is the output of the program.

LINE 1 TRIGGER:
        Memory constructor 0x558f06b050 
LINE 2 TRIGGER:
        onFirstRef on 0x558f06b050
        Count of 0x558f06b050 : 1
LINE 3 TRIGGER:
        Memory constructor 0x558f06b0c0 
        onFirstRef on 0x558f06b0c0
LINE 4 TRIGGER:
        Count of 0x558f06b050 : 2
LINE 5 TRIGGER:
        Count of 0x558f06b050 : 3
LINE 6 TRIGGER:
        Count of 0x558f06b050 : 2
LINE 7 TRIGGER:
        Count of 0x558f06b050 : 1
        Count of 0x558f06b0c0 : 2
LINE 8 TRIGGER:
        Count of 0x558f06b050 : 1
LINE 9 TRIGGER:
        Count of 0x558f06b050 : 2
LINE 10 TRIGGER:
        Count of 0x558f06b050 : 1
        Memory destructor 0x558f06b0c0
        Memory destructor 0x558f06b050
LINE -1 TRIGGER:

In next article, we will look at the problem of circular reference and how to solve it with weak pointer, or wp, as it is called in Android.

Android System Services Patterns

Android has around 100 system services, providing various system functionality and services. Applications run in their own process and communicate with those system services through client class, aka all kinds of Managers, which talks to the service through Binder.

In this article, we will exam Android system service implementation patterns.

1. Client: get a client to access the system services

The most common way to get a client is to call  context.getSystemService(XXXX_SERVICE)
and you will be returned an XxxxManager. Or, be explicit, call context.getPackageManager().
Finally, new a client yourself, e.g new MediaPlayer().

2. Connect : Those XxxxManagers are actually a wrapper of the proxy side of the corresponding service and they always connect to the service with Binder. Following code snippet illustrates the pattern ,assuming XxxxManager is the client of XxxxServices,

    class XxxxManager {
           IXxxxService mXxxxServcie;
           XxxxManager() {
           mXxxxServcie = IXxxxService.Stub.asInterface(
                                                ServiceManager.getService(XXXX_SERVICE_BINDER_NAME));
          }
    }

Where the IXxxxService.Stub.asInterface(ob) simply returns IXxxxxService.Stub.Proxy(obj), which is a proxy side of service. 

3. Services and Server

The services are either hosted in system_server, or one of the native servers, such as mediaserver, surfaceflinger. 

For the services hosted in the system server, there are four varieties. 
  • The service is complete java based, no native code is needed. e.g PackageManagerService.
  • The service has native code, e.g. ComsuerIrService.
  • The service in turn connects to the system daemon with socket, e.g MountService talks to vold.
  • The service connects to another native services, e.g. DisplayManagerService connects to SurfaceFlinger through SurfaceControl class. In this case, DisplayManagerService acts like a client of SurfaceFlinger.
The XxxManager can also connect to the native service directly.
  • CameraManager connects to native CameraService directly through a java wrapper to ServiceManager.
  • SensorManager connects to native SensorServices through the jni. Or, MediaPlayer connects directly to native MediaPlayerService.
Following diagram visualizes what we just talked. 
Android System Services Pattern
Please click here for a complete list of the android system services and their implementation details, including the interface file, client class and server class. Note that this is a unfinished work and I am sure it will never be finished....since the number of services is just too much.. However, once you understand the patterns explained in this article, there is no necessity of checking each and every services. They just look similar and fall into the patterns we talked here. If it is not yet, let me know and I will add it :)

Android Camera2 API Explained

Compared with the old camera API, the Camera2 API introduced in the L is a lot more complex: more than ten classes are involved, calls (almost always) are asynchronized, plus lots of capture controls and meta data that you feel confused about.

No worries. Let me help you out. Whenever facing a complex system need a little bit effort to understand, I usually turns to the UML class diagram to capture the big picture.

So, here is the class diagram for Camera2 API.

Android Camera2 Class Diagram


You are encouraged to read this Android document first and then come back to this article, with your questions. I'll expand what is said there, and list the typical steps of using camera2 API. 

1. Start from CameraManager. We use it to iterate all the cameras that are available in the system, each with a designated cameraId. Using the cameraId, we can get the properties of the specified camera device. Those properties are represented by class CameraCharacteristics. Things like "is it front or back camera", "output resolutions supported" can be queried there.

2. Setup the output targets. The camera image data should always go to somewhere. Use SurfaceView or SurfaceTexture for preview, ImageReader for still picture or MediaRecoder for video recording. All those class have one common element hidden behind  - a Surface. What I means is, for all those classes mentioned, you always can create or get a surface, and, pass that surface to camera, as a capture request target. We'll come back the concept target in the followings steps talking about create a capture session with a list of targets and add a target to capture request.

3. Get a CameraDevice. You call CameraManager.open(cameraId) to get a CameraDevice. Since the call is asynchronized, you will get the CameraDevice in the onOpened() callback.

4. Create a CaptureRequest from the CameraDevice. A CaptureRequest is a immutable package of settings and outputs needed to capture a single image from the camera device. We usually don't create a CaptureRequest directly since that would be error prone and tedious, given the settings for a typical user case, say preview, is almost similar. Builder pattern is applied here. As shown in the class diagram above, a CaptureReques.Builder is created from CameraDevice with a predefined template (createCaptureRequest(CameraDevice.TEMPLATE_PREVIEW)). Then , we use the builder to set up the stuffs that varies. For example, usually you need to call addTarget(Surface) to set up the output targets for this specific CaptureRequest.

5. Create a CaptureRequestSession from the CameraDevice. Consider a CaptureRequestSession as a context in which CaptureRequest will be submitted. To initialize a CaptureRequestSession, you have to provided the initialized Surface as the target surfaces, which were discussed at step 2. Also note that the creation of CaptureRequestSession is also asynchronized. 

6. Submit a CaptureRequest to CaptureRequestSession. Now it is the time to issue the capture request. There are a couple of APIs to do that. Capture() is used to issue a one-time take picture command, while setRepeatingRequest() is used to issue a repeating request for the showing preview. Also, you may want to set various settings, such as the auto focus mode and white balance mode, before submitting the request to start the preview or take a picture.

7. Get the Capture Results. The capture result will get back to you asynchronously through CameraCaptureSession.CaptureCallback. It is the place you can play a sound when the capture starts, or finishes.  The results will approach you piece by piece in CaptureResult on onCaptureProcessed, or you can wait and get them all in TotalCaptureResult on onCaptureComplete.  That is the step 8 in the diagram. And, we're done.

But, wait. Where are the camera picture data?? No worries. They are all good and they are handled by the output target as discussed in step 2. In all cases, the camera is the producer and the targets are the consumer. The camera keeps enqueuing the picture data and once the data is available the consumer will be notified to process them. TextureView will turn the picture data into textures and draw them in the area taken by the TexureView. MediaRecoder will pass the image to codec to compress and then write them out. For ImageReader, it is up to you how to process the data. 

Comments?