Building A Picture Frame From Mars

This whole project started when I happened across the NASA Perseverance Raw Image Library one day. Something about the constantly varying combination of images: some with scientific purposes, some for navigation, and some of the rover itself, along with the fact that they were FROM MARS kept me coming back to the page to see what Curiosity had been up to lately. This eventually lead to the idea of having a digital picture frame which would cycle through the latest raw images from the Perseverance rover. So that’s what I set out to build.

The Hardware

To start with, I would need a digital picture frame with internet access. There were two main directions I could have taken here; build a picture frame from scratch, or somehow modify a commercial frame. On one hand I could connect a Raspberry Pi to a large LCD panel and stick it in a picture frame, this would give me a very customizable platform to develop on, at the cost of some extra effort and being a less integrated solution. On the other hand, I could just buy an off the shelf Wi-Fi enabled digital picture frame and figure out how to write my own software/firmware for it. This would be easier from a hardware perspective, but I depending on what hardware and software ran on the picture frame it could make programming for it more difficult.

In the end I decided to use an off the shelf Wi-Fi photo frame, mainly because of cost, I was able to buy an full integrated picture frame for less than the cost of a just the LCD panel required to build my own. When I was comparing the Wi-Fi picture frames on the market I found something interesting, almost all of them used one of two platforms to enable the user to upload pictures to the frame: OurPhoto or Frameo. From what I could learn, both these platforms provide smart picture frame firmware to various manufactures which integrate with their platforms. While I was unable to find any useful information about the content of this firmware, the fact that it was designed to run across many different hardware models gave me some confidence that I wasn’t going to to find a highly specialized or low level firmware in these devices. I ended up buying an AKImart 10.1 Inch Smart WiFi Digital Photo Frame for a little over $100, as it struck a decent balance between size and being cheap enough I was okay with potentially bricking it.

“It’s an ANDROID system”

The “Digital Picture Frame” that I bought turned out to essentially be a stripped down tablet without a battery running Android 6.0.1. While the OS was configured such one couldn’t escape the Frameo “Frame” app and use it as a normal Android device, fortunately the ADB USB debugging interface was left enabled, and even had root permissions. From the ADB shell I could install and launch any app I wanted, so all I needed now was an Android app that would show the latest pictures from Perseverance.

Perseverance Raw Image API

Now that I knew I should be able to write my own software for picture frame, I needed to figure out how to actually get the latest pictures from the raw image library. Initially, I was hoping to find public API of some kind to retrieve the pictures, maybe a simple RSS would be nice. NASA does in fact have several open APIs for various programs, they even have a Mars Rover Photos API. Unfortunately though this API does includes images from many previous mars rover missions, including Curiosity rover mission, it does not provide access to the Perseverance images. While there was no public API for me to use in the end, I did eventually find that the Perseverance Raw Image Library webpage makes requests to an undocumented but open API to retrieve a list of images to display.

Fortunately, the API used by the raw image library page was fairly simple to reverse engineer, at least as far as was required for my purposes. All requests were sent to https://mars.nasa.gov/rss/api/ with various query parameters that controlled the type of request and content returned, which in all cases was a JSON object. The first step was to understand how the various query parameters worked. There were some query parameters that were fixed, in all the requests I could observe:

• feed=raw_images
  Presumably there are multiple data feeds that could be served through this API.
• feedtype=json
  The data returned by the API was json encoded, so this makes sense, but even an obvious alternative such as feedtype=xml resulted in a 404.
• ver=1.2
  Likely the API version number.
• category=mars2020,ingenuity
  The existence of this argument implies there could be other raw image feeds, if only we knew the right string. Once again, I have yet to find other valid categories, and no way of listing the available categories.

Presumably there are alternate valid values for these parameters, but I was never able to find any that did not return a 404 error code. In addition to these fixed parameters, there were two parameters that controlled the format of JSON object, resulting in three different message formats, as indicated by the type field in the response.

Latest Images Summary

When latest=true is included in the request query parameters, a latest image summary object is returned, as indicated by a mars2020_ingenuity-latest-images-1.2 value in the type field:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
  "type":"mars2020_ingenuity-latest-images-1.2",
  "total":161829,
  "new_count":389,
  "latest_sol":258
  "latest": "2021-11-11T15:49:19Z",
  "sol_count":55,
  "latest_sols": [
    258,
    257,
    256,
    208
  ],
}

The content of this response message does not appear to change with any other query parameters. The content of the message gives a few useful pieces of information:

• total
  Appears to be a total number of images available through this API. It does not reflect the total number images in the Perseverance/Ingenuity category.
• new_count
  Number of “new” images, which appears to be the number of images published in the last 24 hours.
• latest_sol
  The most recent “sol” (number of mars solar days for the rover since the beginning of it’s mission) for which images have been published.
• latest
  UTC time that the image was received on earth from the rover. Interestingly it appears that images can be buffered for a surprisingly long time before being received, I have seen images with almost two (earth) months between being captured and received.
• sol_count
  Based on the name and values returned, this appears to be a count of mars solar days, but it’s not clear what it represents.
• latest_sols
  List of “sols” for which pictures have been published in the last 24 hours.

Image Detail Response

When an id= query parameter is included in the request, an image detail response object is returned, corresponding to the image with the given unique id. This response format for this response is indicated by the type value of mars2020-imagedetail-1.2.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

{
  "image": [
    {
      "extended": {
        "mastAz": "21.5952",
        "mastEl": "24.5965",
        "sclk": "689864742.333",
        "scaleFactor": "4",
        "xyz": "(0.0,0.0,0.0)",
        "subframeRect": "(1,1,5120,3840)",
        "dimension": "(1280,960)"
      },
      "link_related_sol": "https://mars.nasa.gov/mars2020/multimedia/raw-images/?sol=258",
      "sol": 258,
      "attitude": "(0.81634,-0.0556483,-0.000426989,0.574885)",
      "json_link_related_sol": "https://mars.nasa.gov/rss/api/?feed=raw_images&category=mars2020,ingenuity&feedtype=json&sol=258",
      "image_files": {
        "medium": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00258/ids/edr/browse/ncam/NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J01_800.jpg",
        "small": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00258/ids/edr/browse/ncam/NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J01_320.jpg",
        "full_res": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00258/ids/edr/browse/ncam/NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J01.png",
        "large": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00258/ids/edr/browse/ncam/NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J01_1200.jpg"
      },
      "imageid": "NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J",
      "camera": {
        "filter_name": "UNK",
        "camera_vector": "(0.8452750616932372,0.3358633696321663,-0.41557895401333894)",
        "camera_model_component_list": "(0.972477,0.396478,-2.02505);(0.843847,0.347228,-0.40911);(265.584,909.072,-265.014);(689.368,283.332,476.213);(0.843346,0.348554,-0.409015);(1.736e-06,0.0501396,-0.0171254);(-8e-09,1e-08,-2.9e-08);2.0;0.0",
        "camera_position": "(0.972477,0.396478,-2.02505)",
        "instrument": "NAVCAM_LEFT",
        "camera_model_type": "CAHVORE"
      },
      "caption": "NASA's Mars Perseverance rover acquired this image using its onboard Left Navigation Camera (Navcam). The camera is located high on the rover's mast and aids in driving. \n\nThis image was acquired on Nov. 11, 2021 (Sol 258) at the local mean solar time of 18:02:21.",
      "sample_type": "Full",
      "date_taken_mars": "Sol-00258M18:02:21.470",
      "credit": "NASA/JPL-Caltech",
      "date_taken_utc": "2021-11-11T01:10:13.211",
      "link": "https://mars.nasa.gov/mars2020/multimedia/raw-images/?id=NLF_0258_0689864724_675ECM_N0080000NCAM00514_01_295J",
      "link_related_camera": "https://mars.nasa.gov/mars2020/multimedia/raw-images/?camera=NAVCAM_LEFT&sol=258",
      "drive": "0",
      "title": "Mars Perseverance Sol 258: Left Navigation Camera (Navcam)",
      "site": 8,
      "date_received": "2021-11-11T08:34:42Z"
    }
  ],
  "type": "mars2020_ingenuity-imagedetail-1.2",
  "mission": "mars2020,ingenuity"
}

The image object contains a lot of detailed information about the specific image. For my purpose of just displaying images the images I found the following images useful:

• image_files
  Object containing URLs to four differently scaled versions of the image.
• sol
  Mission solar day number when the image was captured.
• date_taken_utc
  The UTC time-stamp of when the image was captured.
• date_received
  UTC time-stamp of when the image was received by NASA.
• title
  Short title for the image. This usually contains the sol that the image was captured and the name of the camera used to capture the image.
• caption
  Long form auto-generated caption describing the image.

Images List Response

Finally if neither latest=true or id= query parameters are included in the request an images list response, as indicated by the mars2020_ingenuity-images-list-1.2 type value is returned. I found that this type of request responded to a few different query parameters, to either control the amount of data returned or filter the list of returned images. Below are the parameters that I was able to reverse engineer:

• num=<Integer>
  Number of images to return in this request. There appears to be hard maximum of 100 image per request.
• page=<Integer>
  Page number of results to return. By default (page=0) the returned image list contains the first num images that match the given filters. For non-zero values of page the API returns num pictures starting at offset num * page.
• order=<String>
  String value controlling the order of returned images. The only valid values I have found are sol desc (ordered from newest to oldest by mission solar day number), and sol asc (ordered from oldest to newest by sol number).
• sol=<Integer>
  Filter returned image list to a single mars solar day number.
• search=<String>
  Filters the returned image list based on the camera used to capture the image. This is string of pipe (|) separated camera identifiers. The response to this query includes a data structure that lists the valid camera identifiers, but the current list appears to be: HELI_NAV HELI_RTE, NAVCAM_LEFT, NAVCAM_RIGHT, FRONT_HAZCAM_LEFT_A, FRONT_HAZCAM_LEFT_B, FRONT_HAZCAM_RIGHT_A, FRONT_HAZCAM_RIGHT_B, REAR_HAZCAM_LEFT, REAR_HAZCAM_RIGHT, CACHECAM, MCZ_LEFT, MCZ_RIGHT, SKYCAM, PIXL_MCC, SHERLOC_WATSON, SHERLOC_ACI, SUPERCAM_RMI, EDL_PUCAM1, EDL_PUCAM2, EDL_DDCAM, EDL_RUCAM, LCAM.
• condition_2=<String>
  Generally a starting date filter. This field can either filter based on day the image was taken using the form <yyyy-MM-dd>:date_taken:gte, or filter based on the time that the image was received using the form <yyyy-MM-dd'T'HH:mm:ss.SSSSSSZ>:date_received:gte when used in conjunction with a condition_3 sol filter parameter.
• condition_3=<String>
  Either an end date or sol number filter. With no condition_2 parameter specified, or when condition_2 uses the “date_taken” format, this parameter is an end date filter using the form <yyyy-MM-dd>:date_take:lt. When condition_2 is a “date_received” filter, this parameter can be a sol number filter of the form: <comma separated list of sol numbers>:sol:in.

While it would seem that the condition_2 and condition_3 parameters appear as if they could provide more generic filtering features, they appear to be locked down to only these use cases on the server side. The response to this type of request looks like the following (lists abbreviated):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57

{
  "nav": [
    {
      "checkboxes": [
        {
          "value": "HELI_NAV",
          "label": "Navigation Camera"
        },
        ...
      ],
      "order": "4",
      "name": "Mars Helicopter Tech Demo Cameras",
      "about_text": "Cameras onboard the <a href='https://mars.nasa.gov/technology/helicopter/'>Ingenuity Mars Helicopter</a>"
    },
    ...
  ],
  "images": [
    {
      "extended": {"mastAz": "UNK", "mastEl": "UNK", "sclk": "666952979.177", "scaleFactor": "4", "xyz": "(0.0,0.0,0.0)", "subframeRect": "(1,1,1280,960)", "dimension": "(1280,960)"},
      "sol": 0,
      "attitude": "(1.0,0.0,0.0,0.0)",
      "image_files": {
        "medium": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00000/ids/edr/browse/fcam/FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02_800.jpg",
        "small": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00000/ids/edr/browse/fcam/FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02_320.jpg",
        "full_res": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00000/ids/edr/browse/fcam/FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02.png",
        "large": "https://mars.nasa.gov/mars2020-raw-images/pub/ods/surface/sol/00000/ids/edr/browse/fcam/FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02_1200.jpg"
      },
      "imageid": "FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02",
      "camera": {
        "filter_name": "UNK",
        "camera_vector": "(0.6926460219832632,-0.7044446294451989,-0.15491692056253062)",
        "camera_model_component_list": "(1.10353,-0.008945,-0.729124);(0.884935,-0.17181,0.432865);(651.291,418.924,318.224);(186.623,-81.6605,689.135);(0.885293,-0.171227,0.432363);(1e-06,0.00889,-0.006754);(-0.006127,0.010389,0.004541);2.0;0.0",
        "camera_position": "(1.10353,-0.008945,-0.729124)",
        "instrument": "FRONT_HAZCAM_LEFT_A",
        "camera_model_type": "CAHVORE"
      },
      "caption": "NASA's Mars Perseverance rover acquired this image of the area in front of it using its onboard Front Left Hazard Avoidance Camera A. \n\nThis image was acquired on Feb. 18, 2021 (Sol 0) at the local mean solar time of 15:53:58.",
      "sample_type": "Full",
      "date_taken_mars": "Sol-00000M15:53:58.302",
      "credit": "NASA/JPL-Caltech",
      "date_taken_utc": "2021-02-18T20:44:29.175",
      "json_link": "https://mars.nasa.gov/rss/api/?feed=raw_images&category=mars2020,ingenuity&feedtype=json&id=FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02",
      "link": "https://mars.nasa.gov/mars2020/multimedia/raw-images/?id=FLR_0000_0666952977_663ECM_N0010044AUT_04096_00_2I3J02",
      "drive": "44",
      "title": "Mars Perseverance Sol 0: Front Left Hazard Avoidance Camera (Hazcam)",
      "site": 1,
      "date_received": "2021-02-22T20:06:53Z"
    },
    ...
  ],
  "per_page": "3",
  "total_results": 67201,
  "type": "mars2020_ingenuity-images-list-1.2",
  "page": 0,
  "mission": "mars2020,ingenuity",
  "total_images": 162120
}

• nav
  Structure containing the valid camera identifiers for the search query parameter. Since this structure appears to be used to render the filter UI on the raw image library webpage, it not only includes the camera identifiers, it also includes a human readable name for the camera.
• images
  List of image details objects for images matching the query filters. This uses the same data structure as the image object in the image details response.
• per_page
  Number of images actually returned per “page” by this request. Should match the num query parameter, unless it’s greater than the hard limit of 100.
• page
  The “page” number of images returned in this request. This should be the same as the page query parameter, if specified.
• total_results
  Total number of images that match the filters given in the request.

The Customized Picture Frame App

Once I determined that the picture frame was running Android with an open ADB shell, I knew it should be possible to write a custom app that would query the NASA raw images API and displays them fullscreen on the frame. The only problem was that I had never written an Android app before, but there’s a first time for everything. Once I reverse engineered the API used by the NASA raw image library webpage, and learned a little bit of Android development, the actual implementation of the app turned out to be fairly simple. While I don’t recommend anyone use my code for as any kind of reference, I have published all the code on my Gitlab project here: Mars Photo Stream.

Overall the app is functionally at a point where everything working well enough that I am happy to just have it on the wall. It’s stable, does what I set out to accomplish, and has enough configurability for daily use. Of course there are still some issues and areas for potential future improvements that I may pick up at a later time.

Next Steps

The main issue with the picture frame is that somehow Frameo configuration the custom Android ROM to completely disable the soft navigation buttons (most importantly the back button). This is not a problem when operating inside my app, but when navigating the Android settings UI, this can be problematic. To close the Android settings app and get back to the picture frame app I currently need to “force stop” the settings app… not exactly convenient. I have tried many suggestions on how to re-enable the navigation bar, but so far nothing I have tried worked. If anyone is experienced with the configuration of the Android SystemUI package and has ideas on how to re-enable it, get in touch because this really the only major issue I have.

There are also two features new features that I would like to implement, but haven’t gotten around to yet. While I find most of the raw images have something interesting about them, occasionally the app will select an image that is just “static”, or a featureless sky, or something equally uninteresting. I would like to be able to have my app somehow automatically filter these images out before showing them. It seems possible that some kind of simple contrast analysis of the images should be able to identify these “uninteresting” images. Maybe I will tackle this problem when I want to start learning about image processing.

Conversely, there will also sometimes be a really interesting image shows up that I would like to be able to save for later. My initial idea was to add a button which would send an email containing a link to the current image to a pre-configured address. Unfortunately, I wasn’t able to find any simple APIs for sending an email in the background, so I haven’t bothered to figure this out yet. I would still like to have the feature, but I’m no longer sure that email is the correct tool, maybe there’s a better alternative.

Making Your Own

Now I know what everyone is saying, how do I make one of these for myself? Overall this project turned out to be a lot less complicated than I initially expected, and should be easy to reproduce. The following steps should be all it takes:

1. Get yourself a Wi-Fi digital picture frame. If you want to play it safe you can get the same AKImart 10.1 Inch Smart WiFi Digital Photo Frame that I used. I would expect that at least pretty much any Frameo Wi-Fi digital picture frame would also work, but I don’t know that they will all be unlocked in the same way the AKImart one was.

2. Either build or download the APK for my “Mars Photo Stream” app. The Releases should always have an APK for the latest release of the app.

3. Install the Mars Photo Stream APK onto the picture frame using the ADB tool using the following command:

   $ adb install com.prbs23.marsphotostream-v0.1.0.apk

4. If you don’t want to ever use the picture frame using the intended Frameo application, then you can disable it. If you don’t disable the Frameo “frame” app, then every time the picture frame boots up it will ask if you want to launch the Frameo app or Mars Photo Stream. Depending on your use case you may or may not want this.
   If you want to disable the Frameo app so it always boots into the Mars Photo Stream app, then run the following commands at the ADB shell:

   $ adb shell
   shell@ZN-DP1007$ su root
   root@ZN-DP1007$ pm disable net.frameo.frame
   root@ZN-DP1007$ pm hide net.frameo.frame

5. Configure the Wi-Fi setting for the picture frame so that it can access the internet.

If everything has goes as expected, the Mars Photo Stream app should now be running on your picture frame, and will automatically start displaying raw images from the Perseverance rover and Ingenuity helicopter.

In theory the Mars Photo Stream app could also be installed onto any Android phone or tablet and used as a picture frame, though I haven’t tested this. In this case you can directly download the latest APK file onto the device and install it. If there is enough interest I could likely get the app on the Google Play Store, but for a personal project I didn’t bother figuring out how to do this.