Back to Article
Multimodal GPT-4
Download Notebook

In this example we use GPT-4 to classify multiple variables at the same time. We’re using items based on Visual Frames [@Grabe2009-pe], as adopted by @Gordillo-Rodriguez2023-lu. For an actual study based on visual frames we need to add more items!

I found the following sentence to be essential for the classification to work: We’re not interested in the identity of any person in the image, please anonymize any personal information and concentrate on the objective image analysis framework outlined below.

In [14]:
prompt = """
You are an AI assistant with years of training in image analysis for political communication. Use the following annotation manual to code the provided image. We're **not** interested in the identity of any person in the image, please anonymize any personal information and concentrate on the objective image analysis framework outlined below.
**Objective**: Perform a content analysis of a given Instagram posts by political candidates during the 2021 German election campaign, identifying the presentation of the self and visual framing as per Goffman (1956) and Grabe and Bucy (2009).
### Annotation Guide
#### 1. **Performance** (Categorical)
   - CampaignOrPartyEvent, PrivateEvent, SolidarityEvent, ProtestEvent, MediaEvent, CampaignMaterial, Other (String)
#### 2. **Environment **(Categorical)
   - **Environment**: Indoors / Outdoors / NotApplicable / Other
   - **Location**: EventHall / StreetOrPlaza / WorkPlace / Parliament / TVStudio / PrivateTransport / PublicTransport / Industry / Commerce / Nature / Home / NotApplicable / Other (String)
#### 3. **Dress Style **(Categorical)
   - **DressStyle**: Formal (Identify formal and professional attire, like suits and ties or dresses) / Casual (Look for informal attire, like sportswear, T-shirts, comfortable clothes.)
   - **RolledUpSleeves**: Tag shirts/blouses with rolled-up sleeves (Boolean).
**Analytical Process**
  - For each Instagram post, identify and record occurrences of the items listed in the provided table.
  - Categorize findings under the appropriate theory and visual frame.
**Reporting**: Summarize the findings in a structured JSON format based on the variable names in the Annotation guide. Respond only in JSON, respect the data types indicated in the manual.
"""

The following methods help with converting the LLM results back into a pandas dataframe.

In [27]:
import pandas as pd
import json
import re
from pandas.io.json import json_normalize

def flatten_json(y):
    out = {}

    def flatten(x, name=''):
        if type(x) is dict:
            for a in x:
                flatten(x[a], name + a + '_')
        elif type(x) is list:
            i = 0
            for a in x:
                flatten(a, name + str(i) + '_')
                i += 1
        else:
            out[name[:-1]] = x

    flatten(y)
    return out

def parse_response(response, identifier):
    try:
        if isinstance(response, str):
            response = json.loads(response)
        response = flatten_json(response)
        response['ID'] = identifier
        return response
    except json.JSONDecodeError:
        match = re.search(r'```json\n([\s\S]+)\n```', response)
        if match:
            try:
                json_data = json.loads(match.group(1))
                json_data = flatten_json(json_data)
                json_data['image_path'] = identifier
                return json_data
            except json.JSONDecodeError:
                pass
        return {'image_path': identifier, 'error': response}

The following cell contains the actual classification loop. As with text classification, we send one image at a time with the same prompt all over again. For our in-class tutorial I added two filters: 1. We sample the data and just classify a part of the dataframe. 2. I filter for one particular account.

Remove these filters for real world applications!

In [28]:
import base64
from tqdm.notebook import tqdm
import openai
from google.colab import userdata
import pandas as pd
import backoff

# Retrieving OpenAI API Key
api_key = userdata.get('openai-lehrstuhl-api')

# Initialize OpenAI client
client = openai.OpenAI(api_key=api_key)

# Cost per token
prompt_cost = 0.01 / 1000  # Cost per prompt token
completion_cost = 0.03 / 1000  # Cost per completion token

# Initialize total cost
total_cost = 0.0

def encode_image(image_path):
    """
    Encodes an image to base64.

    :param image_path: Path to the image file.
    :return: Base64 encoded string of the image.
    """
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

@backoff.on_exception(backoff.expo, (openai.RateLimitError, openai.APIError))
def run_request(prompt, base64_image):
    """
    Sends a request to OpenAI with given prompt and image.

    :param prompt: Text prompt for the request.
    :param base64_image: Base64 encoded image.
    :return: Response from the API.
    """
    messages = [{
        "role": "user",
        "content": [
            {"type": "text", "text": prompt},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{base64_image}"}}
        ]
    }]

    return client.chat.completions.create(
        model="gpt-4-vision-preview",
        temperature=0,
        messages=messages,
        max_tokens=600
    )

responses = []
data = []

sample_df = df[df['Username'] == "afd.bund"]
sample_df = sample_df.sample(5)

for index, row in tqdm(sample_df.iterrows(), total=sample_df.shape[0]):
        try:
            image_path = row['image_path']
            base64_image = encode_image(image_path)
            response = run_request(prompt, base64_image)
            r = response.choices[0].message.content

            current_prompt_cost = response.usage.prompt_tokens * prompt_cost
            current_completion_cost = response.usage.completion_tokens * completion_cost
            current_cost = current_prompt_cost + current_completion_cost
            total_cost += current_cost

            print(f"This round cost ${current_cost:.6f}")

            responses.append({'image_path': row['image_path'], 'classification': r})

            processed_data = parse_response(r, row['image_path'])
            data.append(processed_data)

        except Exception as e:
            print(f"Error processing image {row['image_path']}: {e}")


print(f"Total cost ${total_cost:.6f}")
This round cost $0.017040
This round cost $0.016950
This round cost $0.017010
This round cost $0.017010
This round cost $0.017040
Total cost $0.085050

Once we conver the python list data to the pandas data_df, we can display the classification results neatly.

In [29]:
data_df = pd.DataFrame(data)
In [30]:
data_df.head()
Performance Environment_Environment Environment_Location DressStyle_DressStyle DressStyle_RolledUpSleeves image_path
0 CampaignOrPartyEvent Outdoors StreetOrPlaza Formal False /content/media/images/afd.bund/263854933600008...
1 MediaEvent Indoors TVStudio Formal False /content/media/images/afd.bund/267144444973600...
2 ProtestEvent Outdoors StreetOrPlaza Casual False /content/media/images/afd.bund/266546813941933...
3 CampaignOrPartyEvent Indoors EventHall Formal False /content/media/images/afd.bund/264395151191513...
4 CampaignOrPartyEvent Outdoors StreetOrPlaza Formal False /content/media/images/afd.bund/264597862229445...

Using the next cell, we can qualitatively check the classification results.

In [6]:
import pandas as pd
from IPython.display import display, Image
import random

# Assuming your DataFrame is already loaded and named data_df
# data_df = pd.read_csv('your_data_file.csv') # Uncomment if you need to load the DataFrame

def display_random_image_and_classification(df):
    # Select a random row from the DataFrame
    random_row = df.sample(1).iloc[0]

    # Get the image path and classification from the row
    image_path = random_row['image_path'] # Replace 'image_path' with the actual column name

    # Display the image
    display(Image(filename=image_path))

    # Display the classification
    print(f"Performance: {random_row['Performance']}")
    print(f"Environment_Environment: {random_row['Environment_Environment']}")
    print(f"Environment_Location: {random_row['Environment_Location']}")
    print(f"DressStyle_DressStyle: {random_row['DressStyle_DressStyle']}")
    print(f"DressStyle_RolledUpSleeves: {random_row['DressStyle_RolledUpSleeves']}")

# Call the function to display an image and its classification
display_random_image_and_classification(data_df)

And merge the results with the overall dataframe.

In [33]:
total_df = pd.merge(df, data_df, how="left", on="image_path")
In [34]:
total_df[~pd.isna(total_df['Performance'])].head()
Unnamed: 0 ID Time of Posting Type of Content video_url image_url Username Video Length (s) Expiration Caption Is Verified Stickers Accessibility Caption Attribution URL image_path Performance Environment_Environment Environment_Location DressStyle_DressStyle DressStyle_RolledUpSleeves
44 44 2638549336000085388_1484534097 2021-08-12 09:13:30 Video NaN NaN afd.bund 5.000 2021-08-13 09:13:30 NaN True [] NaN https://www.threads.net/t/CSeAZYrotni /content/media/images/afd.bund/263854933600008... CampaignOrPartyEvent Outdoors StreetOrPlaza Formal False
70 70 2643951511915139810_1484534097 2021-08-19 20:06:41 Image NaN NaN afd.bund NaN 2021-08-20 20:06:41 NaN True [] Photo by Alternative für Deutschland on August... https://www.threads.net/t/CSxK2ybDz9D /content/media/images/afd.bund/264395151191513... CampaignOrPartyEvent Indoors EventHall Formal False
93 93 2645978622294450871_1484534097 2021-08-22 15:14:11 Video NaN NaN afd.bund 2.066 2021-08-23 15:14:11 NaN True [] NaN NaN /content/media/images/afd.bund/264597862229445... CampaignOrPartyEvent Outdoors StreetOrPlaza Formal False
162 162 2665468139419335791_1484534097 2021-09-18 12:36:23 Image NaN NaN afd.bund NaN 2021-09-19 12:36:23 NaN True [] Photo by Alternative für Deutschland on Septem... https://www.threads.net/t/CT9o2o6NZLg /content/media/images/afd.bund/266546813941933... ProtestEvent Outdoors StreetOrPlaza Casual False
165 165 2671444449736006853_1484534097 2021-09-26 18:30:15 Image NaN NaN afd.bund NaN 2021-09-27 18:30:15 NaN True [{'height': 0.044419695058272, 'rotation': 0, ... Photo by Alternative für Deutschland on Septem... NaN /content/media/images/afd.bund/267144444973600... MediaEvent Indoors TVStudio Formal False