I’ve got a lot on my plate…

Introduction

This vignette walks through how to connect to a visual recognition API, and use its food recognition model to send a batch of image files, and retrieve information on the food present in those images.

Prerequisites

You’ll need to install packages (most are listed at the top of the vignette…some aren’t, which is bad practice). You’ll also need an API key.

This vignette is incomplete, in particular it doesn’t currently save the outputs in any kind of useful format.

Inspiration

Based on https://mran.microsoft.com/snapshot/2016-04-12/web/packages/clarifai/vignettes/using_clarifai.html

Nope, that doesn’t work

Base on http://pablobarbera.com/ECPR-SC104/code/08-clarifai-api.html

If you’d rather Python try https://github.com/robmarkcole/Useful-python/blob/master/Clarifai/Clarafai%207-8-2018.ipynb

In theory you could also use https://github.com/cloudyr/RoogleVision or https://github.com/flovv/Roxford to connect to google and microsoft image services respectively, (and no doubt something for Amazon and IBM’s services too), or hook into one of the specialist food APIs (if that’s your focus).

The code

Ok first we’re going to demo getting info via the API for a single image

Load the libraries

#install.packages("devtools")
#devtools::install_github("soodoku/clarifai")
#library(clarifai) NOT WORKING WITH CURRENT VERSION :-(

library(httr)

## Warning: package 'httr' was built under R version 3.5.3

We’re going to load the key file.

source("key_files/clarifai_key.R")
#This is a simple script that contains two lines. Save that script, e.g. in a subfolder key_files/   The 'source' line simply loads the keys.
#clarifai_key <- "string here"
#clarifai_id <- "string here"

Now we authenticate using that

#secret_id(c(clarifai_id, clarifai_key)) 
#get_token()  THIS ISNT WORKING IN CURRENT VERSION

#so we'll do it the other way...
apikey <- clarifai_key #just for easy of copying code from elsewhere we'll rename our variable

base_url <- paste0("https://api.clarifai.com/v2/models/","bd367be194cf45149e75f01d59f77ba7","/outputs") #you'll see the weird string, that's for the food one. You could send it to the face detection model instead a403429f2ddf4b49b307e318f00e528b 
#or the general model https://clarifai.com/models/general-image-recognition-model-aaa03c23b3724a16a56b629203edc62c

Form the requests we’re going to make. Note, this is the image we’re sending…to the food API

#We can either type the object in JSON as text (recommended in this case, given its complexity), or create it from within R as a list

requests <- '
  {
    "inputs": [
      {
        "data": {
          "image": {
            "url": "http://i.imgur.com/XmAr3jV.jpg"
          }
        }
      }
    ]
  }'

#or the list way:
req <- list("inputs" = list())
req$inputs[[1]] <- list(data=list(image=list(url = "http://i.imgur.com/XmAr3jV.jpg")))
requests <- rjson::toJSON(req)

And now let’s POST

r <- POST(base_url, 
    add_headers(
        "Authorization" = paste0("Key ",apikey),
        "Content-Type" = "application/json"),
    body = requests)
r

## Response [https://api.clarifai.com/v2/models/bd367be194cf45149e75f01d59f77ba7/outputs]
##   Date: 2019-04-15 05:13
##   Status: 200
##   Content-Type: application/json; charset=UTF-8
##   Size: 2.22 kB

Ok…so let’s do something with that?

r <- content(r, "parsed")

for (result in r$outputs[[1]]$data$concepts){
    message('object: ', result$name, ' -- probability: ',
        result$value)
}

## object: lamb -- probability: 0.98912746

## object: duck -- probability: 0.84219635

## object: tongue -- probability: 0.77590185

## object: pork -- probability: 0.77310026

## object: sweet -- probability: 0.6828375

## object: chicken -- probability: 0.6412517

## object: beef -- probability: 0.60754395

## object: meat -- probability: 0.5252384

## object: venison -- probability: 0.52096176

## object: milk -- probability: 0.5084926

## object: mutton -- probability: 0.4889681

## object: truffle -- probability: 0.4644618

## object: bird -- probability: 0.46390593

## object: nibble -- probability: 0.45980784

## object: ginger -- probability: 0.4556288

## object: potato -- probability: 0.428442

## object: veal -- probability: 0.40964532

## object: grass -- probability: 0.38871378

## object: curry -- probability: 0.3645004

## object: steak -- probability: 0.35160452

Let’s batch that

Now we want to send a set of images.

First, read a directory of images (from the web or localy)

pic_list <- list.files("food", full.names = T, pattern = ".JPG")
#I would strongly recommend reducing the size of images before proceding!
#There are a bunch of approaches to doing that in R...

If your batch of images are on your machine (rather than on a public folder on the web), you’ll need to convert them to base64. If they’re on the web you can change the code below to use a variant of the url method above.

library(base64enc) #this is bad practice, it should really go at the top

## Warning: package 'base64enc' was built under R version 3.5.2

pic_list_base64 <- lapply(pic_list,base64encode)

You can test if you want

#txt2 <- base64encode("C:/blah/blah/food/IMG_6234.JPG")  #give it the path to your image

#or the list way:
# req <- list("inputs" = list())
# req$inputs[[1]] <- list(data=list(image=list(base64 = txt2)))
# requests <- rjson::toJSON(req)
# 
# 
# r <- POST(base_url, 
#     add_headers(
#         "Authorization" = paste0("Key ",apikey),
#         "Content-Type" = "application/json"),
#     body = requests)
# r
# 
# r <- content(r, "parsed")
# 
# for (result in r$outputs[[1]]$data$concepts){
#     message('object: ', result$name, ' -- probability: ',
#         result$value)
# }

And then go through the same as above basically, two options (the second one is better)

Option one: Go through the list using lapply and send each one separately

whats_that_pic <- lapply(pic_list_base64, function(x){  #what we're doing here is going through the list of base64 images you created with a procedure. You could also create a function to do this.
  req <- list("inputs" = list())
  req$inputs[[1]] <- list(data=list(image=list(base64 = x)))  #'x' is the item in the list we're on
  requests <- rjson::toJSON(req)
  
  r <- POST(base_url, 
            add_headers(
              "Authorization" = paste0("Key ",apikey),
              "Content-Type" = "application/json"),
            body = requests)
  
  r <- content(r, "parsed")
  
  r
  
  for (result in r$outputs[[1]]$data$concepts){
    message('object: ', result$name, ' -- probability: ',
            result$value)
  }
}
)

## object: truffle -- probability: 0.9994509

## object: chocolate -- probability: 0.9974394

## object: sweet -- probability: 0.9622581

## object: candy -- probability: 0.94751525

## object: cookie -- probability: 0.8903085

## object: cake -- probability: 0.8745675

## object: cream -- probability: 0.8616116

## object: coffee -- probability: 0.7683241

## object: milk -- probability: 0.76240355

## object: mushroom -- probability: 0.7337792

## object: potato -- probability: 0.69337285

## object: pastry -- probability: 0.6607497

## object: cocoa -- probability: 0.51976323

## object: bread -- probability: 0.4994815

## object: tuber -- probability: 0.44628826

## object: dessert -- probability: 0.4180743

## object: milk chocolate -- probability: 0.38415602

## object: chips -- probability: 0.38273662

## object: hash -- probability: 0.365521

## object: dough -- probability: 0.35707122

## object: cereal -- probability: 0.9909675

## object: oatmeal -- probability: 0.97471833

## object: granola -- probability: 0.9741415

## object: muesli -- probability: 0.9622045

## object: milk -- probability: 0.92820925

## object: rice -- probability: 0.8862817

## object: corn -- probability: 0.8678268

## object: sweet -- probability: 0.80016446

## object: wheat -- probability: 0.7716304

## object: porridge -- probability: 0.73381054

## object: cinnamon -- probability: 0.72017

## object: oat -- probability: 0.6933167

## object: yogurt -- probability: 0.6463264

## object: coffee -- probability: 0.5903445

## object: banana -- probability: 0.5899875

## object: cornflakes -- probability: 0.57549465

## object: chocolate -- probability: 0.5632621

## object: raisin -- probability: 0.55438405

## object: oatmeal cereal -- probability: 0.5541141

## object: honey -- probability: 0.5110003

## object: banana -- probability: 0.9990096

## object: sweet -- probability: 0.8639921

## object: cream -- probability: 0.8547184

## object: dairy product -- probability: 0.8085454

## object: chocolate -- probability: 0.75613284

## object: strawberry -- probability: 0.7131227

## object: milk -- probability: 0.70548236

## object: peanut butter -- probability: 0.6488902

## object: ice -- probability: 0.6332538

## object: yogurt -- probability: 0.5758665

## object: pudding -- probability: 0.56315506

## object: ice cream -- probability: 0.5539286

## object: peanut -- probability: 0.54354995

## object: butter -- probability: 0.53937244

## object: chips -- probability: 0.4856897

## object: syrup -- probability: 0.4630544

## object: honey -- probability: 0.43911088

## object: berry -- probability: 0.4108279

## object: tapioca -- probability: 0.39465758

## object: dumpling -- probability: 0.34221533

## object: apple -- probability: 0.96392536

## object: sweet -- probability: 0.94115174

## object: vegetable -- probability: 0.93822455

## object: pumpkin -- probability: 0.8627503

## object: pasture -- probability: 0.83511585

## object: pear -- probability: 0.7554394

## object: peach -- probability: 0.748327

## object: melon -- probability: 0.734571

## object: juice -- probability: 0.67565954

## object: egg -- probability: 0.6658032

## object: squash -- probability: 0.6176888

## object: mango -- probability: 0.44059426

## object: nectarine -- probability: 0.43110797

## object: orange -- probability: 0.39466506

## object: potato -- probability: 0.36459038

## object: mochi -- probability: 0.35180628

## object: cantaloupe -- probability: 0.3383726

## object: pepper -- probability: 0.3102156

## object: gourd -- probability: 0.29764217

## object: persimmon -- probability: 0.29236302

## object: pasta -- probability: 0.9907646

## object: vegetable -- probability: 0.866422

## object: noodle -- probability: 0.8506128

## object: spaghetti -- probability: 0.8119204

## object: sauce -- probability: 0.79027545

## object: meat -- probability: 0.7394205

## object: broth -- probability: 0.66396284

## object: soup -- probability: 0.6317607

## object: mushroom -- probability: 0.6186168

## object: beef -- probability: 0.57490665

## object: pea -- probability: 0.5742003

## object: fettuccine -- probability: 0.5588813

## object: tortellini -- probability: 0.5226992

## object: herb -- probability: 0.5147809

## object: legume -- probability: 0.50550234

## object: macaroni -- probability: 0.4863832

## object: pork -- probability: 0.46765995

## object: soy -- probability: 0.45893747

## object: dough -- probability: 0.38240322

## object: ramen -- probability: 0.3741401

## object: salad -- probability: 0.99725556

## object: spinach -- probability: 0.99677837

## object: vegetable -- probability: 0.99643826

## object: herb -- probability: 0.9622885

## object: lettuce -- probability: 0.9589621

## object: arugula -- probability: 0.93880606

## object: basil -- probability: 0.9101343

## object: oil -- probability: 0.7469167

## object: dandelion greens -- probability: 0.69676745

## object: cheese -- probability: 0.6604413

## object: watercress -- probability: 0.6502621

## object: mizuna greens -- probability: 0.57539046

## object: olive oil -- probability: 0.5512258

## object: new zealand spinach -- probability: 0.5382747

## object: miner's lettuce -- probability: 0.46362755

## object: komatsuna -- probability: 0.42024606

## object: summer purslane -- probability: 0.41810292

## object: parmesan -- probability: 0.3799056

## object: tatsoi -- probability: 0.3736297

## object: yao choy -- probability: 0.32308856

## object: cream -- probability: 0.9959474

## object: yogurt -- probability: 0.9945166

## object: milk -- probability: 0.991271

## object: dairy -- probability: 0.9457755

## object: dairy product -- probability: 0.9393966

## object: sour cream -- probability: 0.91296995

## object: white sauce -- probability: 0.8943993

## object: garlic sauce -- probability: 0.8617635

## object: blue cheese -- probability: 0.7913879

## object: sweet -- probability: 0.7192614

## object: cream cheese -- probability: 0.6670171

## object: tzatziki -- probability: 0.6413661

## object: cheese -- probability: 0.59471667

## object: sauce -- probability: 0.58176863

## object: vegetable -- probability: 0.56325936

## object: curd -- probability: 0.47145405

## object: salad -- probability: 0.42687216

## object: coleslaw -- probability: 0.39927113

## object: clam chowder -- probability: 0.39023077

## object: dill -- probability: 0.37979758

## object: chips -- probability: 0.90707266

## object: chicken -- probability: 0.9052212

## object: bacon -- probability: 0.9021294

## object: sauce -- probability: 0.84960234

## object: potato -- probability: 0.8406528

## object: poutine -- probability: 0.8183321

## object: cheese -- probability: 0.8039971

## object: pork -- probability: 0.7929332

## object: cream -- probability: 0.7462302

## object: nachos -- probability: 0.72910583

## object: beer -- probability: 0.6987153

## object: beef -- probability: 0.61058277

## object: corn -- probability: 0.6105454

## object: steak -- probability: 0.60894316

## object: crab -- probability: 0.60693395

## object: sweet -- probability: 0.59788543

## object: chili -- probability: 0.59686714

## object: tacos -- probability: 0.5758448

## object: lobster -- probability: 0.57215196

## object: lamb -- probability: 0.549315

## object: banana -- probability: 0.9302554

## object: butter -- probability: 0.8768802

## object: peanut -- probability: 0.85372376

## object: chicken -- probability: 0.76398027

## object: gnocchi -- probability: 0.7404165

## object: peanut butter -- probability: 0.6769316

## object: potato -- probability: 0.6560601

## object: coconut -- probability: 0.65582573

## object: rice -- probability: 0.6216823

## object: cream -- probability: 0.5761461

## object: oatmeal -- probability: 0.5478167

## object: cheese -- probability: 0.53859895

## object: chips -- probability: 0.5125718

## object: sweet -- probability: 0.5103579

## object: sauce -- probability: 0.50668883

## object: garlic -- probability: 0.44809264

## object: sausage -- probability: 0.44446173

## object: chocolate -- probability: 0.4326467

## object: cashew -- probability: 0.4278672

## object: milk -- probability: 0.41721702

#we want that to save everything to a list, but it isn't yet...must remember how to do that

Option two: More sensible, write a single query that sends a bunch of images in batch https://clarifai.com/developer/guide/inputs#inputs

#You can also send multiple items (I think it said ~100) at a time.
#We're going to use a function to do this, and add image ID (the file name)
#if you want to test withou using the long base64, you can use e.g. these lists
#x <- list("one","two","three")
#y <- list("IDa","IDb","IDc")
#all the function below is doing is adding to the list in the following form:
#req$inputs[[1]] <- list(data=list(image=list(base64 = c("one"))),id = ("IDa"))

#note this isn't quite right, but it does work
map_to_JSON <- function(base_list,name_list,input_list) {
  x <- length(input_list$inputs) + 1
  req$inputs[[x]] <<- list(list(data=list(image=list(base64 = c(base_list))),id = (name_list)))
}

req <- list("inputs" = list()) #create the empty list again
#Then, run the function. Again, if yo uwant to test it using the short lists:
#req$inputs <- mapply(map_to_JSON, x,y,req)
#requests <- rjson::toJSON(req)
#requests

req$inputs <- mapply(map_to_JSON, pic_list_base64,pic_list,req)
requests <- rjson::toJSON(req)


r <- POST(base_url, 
          add_headers(
            "Authorization" = paste0("Key ",apikey),
              "Content-Type" = "application/json"),
              body = requests)
r <- content(r, "parsed")
  
for (result in r$outputs[[1]]$data$concepts){
    message('object: ', result$name, ' -- probability: ',
            result$value)
}

## object: truffle -- probability: 0.9994514

## object: chocolate -- probability: 0.99744016

## object: sweet -- probability: 0.9622659

## object: candy -- probability: 0.94752854

## object: cookie -- probability: 0.8902824

## object: cake -- probability: 0.87455505

## object: cream -- probability: 0.86158705

## object: coffee -- probability: 0.7683189

## object: milk -- probability: 0.76243407

## object: mushroom -- probability: 0.73383385

## object: potato -- probability: 0.69336736

## object: pastry -- probability: 0.6607301

## object: cocoa -- probability: 0.5197296

## object: bread -- probability: 0.49942887

## object: tuber -- probability: 0.4463967

## object: dessert -- probability: 0.4179826

## object: milk chocolate -- probability: 0.38418543

## object: chips -- probability: 0.3826905

## object: hash -- probability: 0.36544794

## object: dough -- probability: 0.35702986

#you'll have to work out how to navigate the r variable now. 
#you'll see e.. if you do r$outputs[[2]] you should get evvverrryytthhing for the second item in the list of pics you sent, you can check r$outputs[[9]]$input$`id`

How nutritious is my plate?

Ok, then let’s get the nutrition info for one of those images

Inspiration

https://cran.r-project.org/web/packages/NutrienTrackeR/index.html https://github.com/MLH/clarifai-food-nutrition-demo https://stories.mlh.io/watch-what-you-eat-with-clarifai-71185734bc61

Check: https://cran.r-project.org/web/packages/NutrienTrackeR/vignettes/NutrienTrackeR.html looks useful You could also use https://github.com/openfoodfacts/openfoodfacts-python

The code

#load a simple key file with a single line
#myWolframAppId <- "xxx"
#source("key_files/wolfram_key.R")  hm. nope, have to pay for that now?
library(NutrienTrackeR) #again, ideally do this at top of doc

## Warning: package 'NutrienTrackeR' was built under R version 3.5.3

#and then use varients off this query
findFoodName(keywords = c("Tomato", "raw"), food_database = "USDA")

##                                          11527 
##                         "Tomatoes, green, raw" 
##                                          11529 
## "Tomatoes, red, ripe, raw, year round average" 
##                                          11695 
##                        "Tomatoes, orange, raw" 
##                                          11696 
##                        "Tomatoes, yellow, raw"

#it happens that this works on the list, they should all be structured the same but this isn't a good way to access it...someone else can work this out
top_food_in_pic <- r$outputs[[1]][[6]][[1]][[1]][[2]][1]

findFoodName(keywords = c(top_food_in_pic), food_database = "USDA")

##                                      7942 
##                    "Pate, truffle flavor" 
##                                     19138 
## "Candies, truffles, prepared-from-recipe"

USDA <- as.data.frame(food_composition_data$USDA)
subset(USDA, food_id == 17370)