Damian Mehers' Blog Android, VR and Wearables from Geneva, Switzerland.

20Jan/161

Radical surgery: Slimming Pebble apps down to run on Aplite

A long way to go

In December 2015, when first I released Powernoter, an unofficial Evernote client for the Pebble Watch, I initially targeted Pebble Time (codename Basalt), and Pebble Time Round (codename Chalk).

After all there was already the official Evernote Pebble app (which I also created) for the original Pebble (codename Aplite).

Then Pebble released a firmware update and SDK for the original Pebble which meant that I could easily release Powernoter for the original Pebble too, using the same SDK I'd already used.

This is the build log from the first time I built Powernoter targeting Aplite (the original Pebble), Basalt (Pebble Time) and Chalk (Pebble Time Round):

-------------------------------------------------------
BASALT APP MEMORY USAGE
Total size of resources:        26461 bytes / 256KB
Total footprint in RAM:         25895 bytes / 64KB
Free RAM available (heap):      39641 bytes
------------------------------------------------------- 
...
-------------------------------------------------------
CHALK APP MEMORY USAGE
Total size of resources:        26461 bytes / 256KB
Total footprint in RAM:         25943 bytes / 64KB
Free RAM available (heap):      39593 bytes
------------------------------------------------------- 
...
-------------------------------------------------------
APLITE APP MEMORY USAGE
Total size of resources:        26341 bytes / 125KB
Total footprint in RAM:         23789 bytes / 24KB
Free RAM available (heap):      787 bytes
------------------------------------------------------- 

See the 787 bytes on the last line? That was how much free memory my app had before it even started running on an original Pebble. Before it created its first window or allocated memory to receive and send messages.

Although I successfully built Powernoter for Aplite, it couldn't even start up, crashing immediately as it ran out of memory.

Not so verbose with the error messages

The first thing I did, was to run the pebble analyze-size command, which gave me a sense of where the memory was being used.

Like all good programmers, I very carefully and very consistently checked all OS calls for out of memory situations, and logged (very) verbose messages if I ran out of memory. Like this:

  bitmap_layer = bitmap_layer_create(image_layer_size);
  if(!bitmap_layer) {
    APP_LOG(APP_LOG_LEVEL_ERROR, "Couldn't allocate memory for the image");
    ...

All those strings had to be allocated somewhere. I went through my app and removed all those lovely descriptive messages. Instead I just logged the line number - that was enough to work out where it went wrong.

  bitmap_layer = bitmap_layer_create(image_layer_size);
  if(!bitmap_layer) {
    OOMCF();
    ...

I defined a couple of macros for Out Of Memory (OOM) situations:

#define OOM(s) log_oom(__FILE_NAME__, __LINE__, (int)s)
#define OOMCF() log_create_failed(__FILE_NAME__, __LINE__)
void log_create_failed(char* file, int line) {
  app_log(APP_LOG_LEVEL_DEBUG, file, line, "create failed %d free", (int)heap_bytes_free());
}

void log_oom(char* file, int line, int size) {
  app_log(APP_LOG_LEVEL_DEBUG, file, line, "oom %d, %d", size, (int)heap_bytes_free());
}

I also declared some handy logging macros, so that debug log strings were stripped out of shipping builds

#ifdef SHIPPING
#define LOG_MEM_START()
#define LOG_MEM_END()
#define LOG_FUNC_START(name)
#define LOG_FUNC_END(name)
#define LOG_DBG(fmt, args...)
#define LOG_ERR(fmt, args...) app_log(APP_LOG_LEVEL_ERROR, __FILE_NAME__, __LINE__, " ")
#else
#define LOG_DBG(fmt, args...) app_log(APP_LOG_LEVEL_DEBUG, __FILE_NAME__, __LINE__, fmt, ## args)
#define LOG_MEM_START() app_log(APP_LOG_LEVEL_DEBUG, __FILE_NAME__, __LINE__, "start %d", (int)heap_bytes_free())
#define LOG_MEM_END() app_log(APP_LOG_LEVEL_DEBUG, __FILE_NAME__, __LINE__, "end %d", (int)heap_bytes_free())
#define LOG_FUNC_START(name) app_log(APP_LOG_LEVEL_DEBUG, __FILE_NAME__, __LINE__, "%s invoked", name)
#define LOG_FUNC_END(name) app_log(APP_LOG_LEVEL_DEBUG, __FILE_NAME__, __LINE__, "%s returning", name)
#define LOG_ERR(fmt, args...) app_log(APP_LOG_LEVEL_ERROR, __FILE_NAME__, __LINE__, fmt, ## args)
#endif

Use statics in moderation

Next I looked into how I was defining static variables. I like statics because they are only visible to the file in which they are declared: a primitive form of encapsulation. A typical C source file might have started with:

static CustomMenu* customMenu;
static CustomMenuItem* items;
static uint16_t itemCount;
static AppTimer *send_timeout_timer;
static NoteSelectedCallback noteSelectedCallback;

The types don't matter (CustomMenu is my own class that does things like automatically scrolling long menu items).

What matters is that I have four pointers and a short declared as statics, meaning I have a whole chunk of memory statically allocated just for this one file.

Powernoter is not a small app ... this multiplied by tens of files means that I had a load of memory statically allocated, which was never used unless the user was actually invoking the functionality represented by those files.

The solution was to move to a dynamically allocated memory:

typedef struct NoteList {
  CustomMenu *customMenu;
  CustomMenuItem *items;
  uint16_t itemCount;
  AppTimer *send_timeout_timer;
  NoteSelectedCallback noteSelectedCallback;
} NoteList;

I only allocate a NoteList when it is being used, and free it as soon as possible.

Omit needless code

Although the SDK includes definitions for things like DictationSession on Aplite, so that code can be compiled regardless of the platform (you do need to check return calls though), it made no sense to include that code at all. I #ifdefed whole chunks of code to reduce the app size:

#ifdef SUPPORTS_VOICE
static void dictation_session_callback(DictationSession *session, DictationSessionStatus status,
                                       char *transcription, void *context) {
  LOG_FUNC_START("dictation_session_callback");
  if(DictationSessionStatusSuccess == status) {
    if(!noteContext->waitingAnimation) {
      if(noteContext->customMenu) {
        layer_set_hidden(custom_menu_get_layer(noteContext->customMenu), true);
      }
...
}
#endif

SUPPORTS_VOICE is my own macro:

#ifndef PBL_PLATFORM_APLITE
#define SUPPORTS_VOICE
#else
#define LOW_MEMORY_DEVICE
#endif

Pebble have added a PBL_MICROPHONE macro so my use of SUPPORTS_VOICE is no longer necessary.

I did the same thing for animations and color support.

Although I think I am a decent enough software engineer, I am under few illusions as to my abilities as a designer, which is why I let you choose your very own foreground and background colors in Powernoter, except if you are running on an original Pebble, in which case all that code, including the color names, is #ifdefed out.

Be careful what you ask for (when calling app_message_xyz_maximum)

Once upon a time were were limited to 120 or so bytes per message sent between the watch and the phone. I wrote inordinately complex code to page menu items in dynamically from the phone to the watch so that you could scroll through infinitely long menus. Then Pebble gave us what we wanted, with massive (8Kish) message buffers.

When you only have a little memory free to start with, the last thing you want to do is go allocating 8K buffers. It won't work.

My code to determine the size of the input buffer looks like this now:

#ifdef LOW_MEMORY_DEVICE
#define MAX_INBOX_SIZE 512
#else
#define MAX_INBOX_SIZE 4096
#endif

The LOW_MEMORY_DEVICE macro is set on Aplite only. Users on the original Pebble won't see an enormous number of notes listed, or a lot of a note's content, but at least they'll see something.

Make long strings into Resources

There is an excellent Internationalization sample for the Pebble. Although Powernoter isn't internationalized, there are no strings hardcoded in code ... all strings are accessed via a single point. I include the strings in a single source file in the app, except for certain very long strings, such as the About page. These I load as resources from files:

static char* loadResource(uint32_t resourceId) {
  ResHandle handle = resource_get_handle(resourceId);
  size_t res_size = resource_size(handle);

  // Copy to buffer
  char* result = (char*)malloc(res_size + 1);
  if(!result) {
    OOM(res_size);
    result = (char*)malloc(1);
    if(result) {
      *result = '\0';
    }
    return result;
  }
  resource_load(handle, (uint8_t*)result, res_size);
  result[res_size] = '\0';
  return result;
}

Once I'm done with them, I free them as quickly as possible.

Summary

In case you were wondering, this is how things look right now:

CHALK APP MEMORY USAGE
Total size of resources:        27313 bytes / 256KB
Total footprint in RAM:         24244 bytes / 64KB
Free RAM available (heap):      41292 bytes
-------------------------------------------------------
BASALT APP MEMORY USAGE
Total size of resources:        27313 bytes / 256KB
Total footprint in RAM:         24176 bytes / 64KB
Free RAM available (heap):      41360 bytes
-------------------------------------------------------
APLITE APP MEMORY USAGE
Total size of resources:        13966 bytes / 125KB
Total footprint in RAM:         17353 bytes / 24KB
Free RAM available (heap):      7223 bytes
------------------------------------------------------- 

Getting from 787 bytes free to 7,223 bytes free, so that Powernoter can really run on Aplite involved many changes, some which I'd say were generally good practice (reducing statics and instead using structs which are allocated/freed), and some less so (removing error log messages).

In general I don't think the code looks too unreadable as a result of supporting Aplite ... certainly I'd prefer not to have as many #ifdefs sprinkled throughout my code as I have, but it's not that bad.

You may also wish to check out this Pebble presentation on Pebble app memory usage.

One thing is for sure, the changes I had to make to Powernoter to get it to run on Aplite are nothing compared with the miracles the Pebble team pulled to get the original Pebble to support the same SDK as Pebble Time and Pebble Time Round.

About me

I'm an independent consultant and speaker, available for ad-hoc Pebble, Android and Android Wear and Tizen consulting and development.

If you like and use Powernoter, please consider supporting it.

On the other hand if something is missing or doesn't work, check out this Trello board where you can comment to request enhancements or report bugs.

Filed under: Pebble, Wearables 1 Comment
23Aug/157

Using Android Wear to control Google Cardboard Unity VR

Using a VR headset, even one as simple as Google Cardboard, can be mind-blowing.  Nevertheless it is the little things that can also be disconcerting.  For example looking down and seeing you have no arms, despite the fact they still very much feel as though they exist.

I’m convinced that VR experiences are going to transform not just games, but interaction with computers in general, and I’ve been experimenting with some ideas I have about how to create truly useful VR experiences.

As I was working to implement one of my ideas, it occurred to me that I might be able to use the orientation sensors in the Android Wear device I was wearing.  Why not use them as input into the VR experience I was creating?  What if I could bring part of my body from the real world into the VR world?  How about an arm?

I decided to try to find out, and this was the answer:

The experience is nowhere near good enough for games.  But I don’t care about games.  I want to create genuinely useful VR experiences for interacting with computers in general, and I think this is good enough.  I can point to objects, and have them light up.  I can wear smart watches on both wrists (because I really am that cool) and have two arms available in the VR world. 

By tapping and swiping on the wearable screens I can activate in-world functionality, without being taken out of it.  It sure beats sliding a magnet on the side of my face, because it is my arm I am seeing moving in the virtual world.

In the rest of this article I’m going to describe some of technical challenges behind implementing this, how I overcame them, and some of the resources I used on the way.

The tools

This is part of my workspace: Android Studio on the left, Unity on the top-right and MonoDevelop on the bottom-left:

my workspace

I had many reference browser windows open on other screens (obviously), and creating this solution required me being very comfortable in Android, Java and C#.  I’m relatively new to Unity.

Creating a Unity Java Plugin by overriding the Google Cardboard Plugin

The Unity Android Plugin documentation describes how you can create plugins by extending the UnityPlayerActivity Java class, and I experimented with this a little.  I created an Android Library using Android Studio, and implemented my own UnityPlayerActivity derived class.

After a little hassle, I discovered that Unity now supports the “aar” files generated when compiling libraries in Android Studio, although I found the documentation a little out of date on the matter in places.  It was simply a question of copying my generated “aar” file into Unity under Assets|Plugins|Android

image

image

When it came to a Google Cardboard Unity project, what I discovered though, is that Google had got there first.  They had created their own UnityPlayerActivity called GoogleUnityActivity.  What I needed to do was override Google’s override:

image

I included Google’s unity classes as dependencies in my library project:

image

Once I’d copied the aar file into the Unity Android Plugins folder and ran the test app, I was delighted to see my activity say “Cooey” in the log.

image

Receiving the watch’s orientation to the phone

The next step was to receive Android Wear Messages from Android Wear on the watch, containing orientation messages.

I recreated my project, this time including support for Android Wear:

image

I made the Unity activity I’d created do a little more than say “Cooey”. 

First I used the Capabilities mechanism to tell other Android Wear devices that this device (the phone) was interested in arm orientation messages:

image

… and I set it up to receive Android Wear messages and pass them over to Unity using UnitySendMessage:

image

Sending the watch’s orientation to the phone

This was simply a question of looking out for Android Wear nodes that supported the right capability, listening for orientation sensor changes, and sending Android Wear messages to the right node.  This is the watch code:

image

I did discover that some wearables don’t support the required sensors, although I imagine more modern ones will.

Using the watch’s orientation to animate a block on the screen

Inside Unity I created a cube which tweaked into a rectangle, and made it a child of the CardboardMain’s camera, so that it moved when I moved:

image

See the “Script” field on the bottom right-hand side?  I have a script called “WristController” that is attached to the “wrist” (white blob).  This is where I receive orientation messages sent from the watch, via the UnityPlayerActivity derived Java class I’d created.

I started off simply assigning the received orientation to the block’s orientation by assigning to transform.eulerAngles

image

This worked, but was super-jerky.  I went searching and discovered Lerps and Slerps for smoothly moving from one rotation to another.  My updated code:

image

Animating an arm instead of a block

I was pleased to be smoothly animating a block, but my arm doesn’t quite look like that.  It is more armish.  I went looking for a model of an arm that I could import and use instead.  I found a YouTube Unity video called ADDING ARMS by Asbjørn Thirslund, in which he explains how to to import and use a free arms model by eafg.

It was simply a question of sizing and positioning the arms properly as a child of the Cardboard main camera, and then adding the script I’d used to animate the block.

I also removed the right-hand arm, since it looked a little weird to have a zombie arm doing nothing.

image

The ArmController script you see in this screen capture has the same contents as the WristController I’d used to move the block.

Final Thoughts

There is enough of a lag to make this technique impractical for games, but not enough to make it impractical for the kinds of non-game experiences I have in mind. 

I’d also need to add calibration, since the watch may be pointing in any direction initially – if I assume it always starts straight out, that would be good enough.  Detecting where the arm is pointing shouldn’t be too hard, since the cardboard code already does gaze detection – so many possibilities, but so little time for side-projects such as this!

This has been a fun interlude on my way to creating what I hope to be a genuinely useful VR experience based around browsing books your friends have read … more on that later.

16Mar/150

On Pulse: Why your basal ganglia and wearables were made for each other

I just posted Why your basal ganglia and wearables were made for each other

Filed under: Wearables No Comments
22Jan/151

Scrolling long Pebble menu items

This is a technical blog post.  Warning: contains code.

We recently pushed version 1.2 of Evernote for the Pebble to the Pebble App Store.  It is a minor release, with one bug fix, and one new feature.

The bug fix is related to support for the additional character sets that Pebble can now display.

The enhancement is what this blog post is about.  Since we released the first version of the app, which was generally well received, we’ve received emails from people complaining that their note titles, notebook names, tag names etc. don’t fit on the Pebble screen.  They are cut off, and hard to read.  People asked if we could make menu items scroll horizontally if they didn’t fit.

My response was generally something along the lines of “sorry, but we use the Pebble’s built-in menuing system, and until they support scrolling menu items horizontally, we can’t do anything”.  I never felt great about this response, but it was the genuine situation.  However before I pushed the 1.2 release with the character-set bug-fix, I thought I’d take a look at scrolling the menu items.  Turns out, it was surprisingly easy.

You can see what I’m talking about here:

 

The funny thing about the Evernote Pebble watch app is that it knows almost nothing about Evernote.  The Evernote intelligence is all delegated to the companion app that runs on the Phone.  The watch app knows how to display massive menus (paging items in and out as necessary), checkboxes, images, text etc. 

When the user scrolls to a new menu item, we kick off a wait timer using app_timer_register waiting for one second.  If the user scrolls to another menu item before the timer has expired, we wait for a new second, this time using app_timer_reschedule:

static void selection_changed_callback(Layer *cell_layer, MenuIndex new_index, MenuIndex old_index, 
void *data) {
WindowData* window_data = (WindowData*)data;
window_data->moving_forwards_in_menu = new_index.row >= old_index.row;
if(!window_data->menu_reloading_to_scroll) {
initiate_menu_scroll_timer(window_data);
} else {
window_data->menu_reloading_to_scroll = false;
}
}

The above method is called by the Pebble framework when the user scrolls to a new menu item.  The check for menu_reloading_to_scroll is called to work around some behavior I’ve seen.  This callback invokes the following method:

static void initiate_menu_scroll_timer(WindowData* window_data) {
// If there is already a timer then reschedule it, otherwise create one
bool need_to_create_timer = true;
window_data->scrolling_still_required = true;
window_data->menu_scroll_offset = 0;
window_data->menu_reloading_to_scroll = false;
if(window_data->menu_scroll_timer) {
// APP_LOG(APP_LOG_LEVEL_DEBUG, "Rescheduling timer");
need_to_create_timer = !app_timer_reschedule(window_data->menu_scroll_timer,
SCROLL_MENU_ITEM_WAIT_TIMER);
}
if(need_to_create_timer) {
// APP_LOG(APP_LOG_LEVEL_DEBUG, "Creating timer");
window_data->menu_scroll_timer = app_timer_register(SCROLL_MENU_ITEM_WAIT_TIMER,
scroll_menu_callback, window_data);
}
}

As you can see it uses a WindowsData structure, which is a custom structure associated with the current window via window_set_user_data.  Once the timer expires it calls scroll_menu_callback:

static void scroll_menu_callback(void* data) {
WindowData* window_data = (WindowData*)data;
if(!window_data->menu) {
return;
}
window_data->menu_scroll_timer = NULL;
window_data->menu_scroll_offset++;
if(!window_data->scrolling_still_required) {
return;
}

// Redraw the menu with this scroll offset
MenuIndex menuIndex = menu_layer_get_selected_index(window_data->menu);
if(menuIndex.row != 0) {
window_data->menu_reloading_to_scroll = true;
}
window_data->scrolling_still_required = false;
menu_layer_reload_data(window_data->menu);
window_data->menu_scroll_timer = app_timer_register(SCROLL_MENU_ITEM_TIMER, scroll_menu_callback,
window_data);
}

This code is called once when the timer initiated by initiate_scroll_menu_timer expires (after the one second delay), and then it invokes itself repeatedly using a shorter delay (a fifth of a second), until the menu item is fully scrolled.  The call to menu_layer_reload_data is what causes the menu to be redrawn, using the menu_scroll_offset to indicate how much to scroll the text by.

This is the method that gets called by the draw_row_callback to get the text to be displayed for each menu item:

void get_menu_text(WindowData* window_data, int index, char** text, char** subtext) {
MenuItem* menu_item = getMenuItem(window_data, index);
*text = menu_item ? menu_item->text : NULL;
*subtext = menu_item && menu_item->flags & ITEM_FLAG_TWO_LINER ?
menu_item->text + strlen(menu_item->text) + 1 : NULL;
if(*subtext != NULL && strlen(*subtext) == 0) {
*subtext = NULL;
}

MenuIndex menuIndex = menu_layer_get_selected_index(window_data->menu);
if(*text && menuIndex.row == index) {
int len = strlen(*text);
if(len - MENU_CHARS_VISIBLE - window_data->menu_scroll_offset > 0) {
*text += window_data->menu_scroll_offset;
window_data->scrolling_still_required = true;
}
}
}

The bolded code “scrolls” the text if the row corresponds to the currently selected item by indexing into the text to be displayed, and indicating that scrolling is still required.  I’m not happy with using the fixed size MENU_CHARS_VISIBLE to decide whether or not to scroll – it would be much nicer to measure the text and see if it fits.  If you know of a simple way to do this please comment!

The final thing I needed to do was to actually send longer menu item text from the phone to the watch.  Since Pebble now support sending more than 120 or so bytes this was much easier.  I’m sending up to 32 characters now.

In summary I’m simply using a timer to redisplay the menu, each time scrolling the current menu item’s text by indexing into the character array, and I stop the timer once it has all been displayed.

Filed under: Pebble, Wearables 1 Comment
12Jan/142

Sony SmartWatch2 scrollable text

Sony have adopted an intriguing approach to development for their SmartWatch 2.  Unlike other Smart Watches you don’t write code that runs on the watch itself.  Instead all your code runs on an Android phone.  You define the watch UI using standard Android layouts, and that UI is remoted onto the watch.

Your app can respond to events such as touches, since these events are sent from the watch to your phone, and then delivered to your app.

This is kind of cool, in that you don’t have to debug on the watch.  It is simpler, and I think it works well for relatively simple apps.

There are, however limitations on the UI elements that you can display.  Lists work well, but it isn’t currently possible to create a scrollable text area.

For the experimental app I was working on, this was a big issue. I needed to display text that went on for more than one screen.

I eventually found a way around this restriction.  I render the text into a bitmap in memory on the phone, then I split the bitmap up into watch-screen sized chunks, and I use each chunk as an element in a list.  This works.  You can scroll through your text, albeit a page at a time.

My list is derived from ManagedControlExtension, and in the onResume I render the text to a bitmap member variable by calling renderTextToCanvas:

  private void renderTextToCanvas() {
    mBitmap = Bitmap.createBitmap(mScreenWidth, mScreenHeight * SCREEN_PAGES, Bitmap.Config.ARGB_8888);
    mBitmap.setDensity(DisplayMetrics.DENSITY_DEFAULT);
    mCanvas = new Canvas(mBitmap);
    mCanvas.setDensity(DisplayMetrics.DENSITY_DEFAULT);

    TextPaint tp = new TextPaint();
    tp.setColor(Color.WHITE);
    tp.setTextSize(18);

    String text = mNote.textContent;

    if(text == null) {
      Log.d(TAG, "Empty text ...");
      text = mContext.getString(R.string.empty_note);
      tp.setTextSkewX(-0.25f); // Italics
    }

    StaticLayout sl = new StaticLayout(text, tp, mScreenWidth, Layout.Alignment.ALIGN_NORMAL, 1.2f,
                                       0f, false);

    mCanvas.save();
    sl.draw(mCanvas);
    mCanvas.restore();
  }

Then when a list element is requested I render the appropriate bitmap chunk and return it:

@Override
  public void onRequestListItem(final int layoutReference, final int listItemPosition) {
    Log.d(TAG, "onRequestListItem() - position " + listItemPosition);
    if (layoutReference != -1 && listItemPosition != -1 && layoutReference == R.id.listView) {
      ControlListItem item = createControlListItem(listItemPosition);
      if (item != null) {
        sendListItem(item);
      }
    }
  }

  protected ControlListItem createControlListItem(int position) {
    Bitmap bitmap = Bitmap.createBitmap(mBitmap, 0, mScreenHeight * position,
                                                    mScreenWidth, mScreenHeight);
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    bitmap.compress(Bitmap.CompressFormat.PNG, 100, byteArrayOutputStream);

    ControlListItem item = new ControlListItem();
    item.layoutReference = R.id.listView;
    item.dataXmlLayout = R.layout.note_content_item;
    item.listItemPosition = position;
    item.listItemId = position;

    Bundle imageBundle = new Bundle();
    imageBundle.putInt(Control.Intents.EXTRA_LAYOUT_REFERENCE, R.id.imageView);
    imageBundle.putByteArray(Control.Intents.EXTRA_DATA, byteArrayOutputStream.toByteArray());

    item.layoutData = new Bundle[] { imageBundle };

    return item;
  }

All this leads to scrollable text:

Filed under: Wearables 2 Comments
12Jan/141

So, how did you die? Wearables as the human black box.

Yesterday I gave a presentation at the excellent Mobile Central Europe conference in Warsaw, Poland, on Evernote and wearable devices.

When talking about the convergence of activity monitoring devices and smart watches I voiced a sudden thought.

How long will it be before someone dies of a heart-attack, and we use the health-sensors that are being incorporated into smart wearable devices to at what was happening the moments before they died?

We might then look for the same signs in others and warn them to “Lie down!, lie down!, medical assistance is on its way” … the human equivalent of “terrain!, terain!, pull up” in the cockpit.

File:Flightrecorder.jpg

Filed under: Wearables 1 Comment