Time is the thing: AI-powered, occasionally incorrect calendar events

Using an LLM to manage a Google Calendar, with multimodal GPT4, scotch tape, URL encoding and screenshots

I live and die by my Google Calendar. The first event is from 9 May 2007 (“Meet M.”). Since then, with few exceptions, every day is pretty much accounted for.

My pet peeve is how much time it takes to add events. Especially if you want to have a nice descriptive title, correct hours and location for every event, it takes at least 2 minutes to enter all that. A tragic burden, I know.

ChatGPT in principle doesn't have a Google Calendar integration, but we'll find a way to make it work!

Schedule what I tell you

Let's go! First we'll try asking ChatGPT to put an event on my Google Calendar.

No dice, there's no calendar integration built-in.

But, Google Calendar has a nice feature that constructs new events from special links that encode all of the event information in URL parameters. ChatGPT cannot add an event to my calendar, but it could craft the right link that will pre-populate all the information. Let’s try!

And just like this, it worked! The “Google Calendar Event Link” is a clickable URL that points to google.com/calendar, with the event details encoded in request parameters:

text=Dentist+Appointment&details=Visit+to+the+dentist.&location=27+Rue+de+Mogador

Let’s try clicking on it

Almost perfect! The time is wrong due to time zones issues, but even this only takes one follow-up request to correct:

The new link works correctly 🥳.

Schedule what I show you

OK, scheduling from a text description is nice, but often the input is visual: an event website or physical leaflet. Can it handle scheduling based on information in a picture?

Here’s an event page of an amazing cinema club in Paris:

Let’s give it a try:

The link works perfectly 👏. GPT4 is a multimodal model, it can handle requests that are explained in text but require information contained in a picture.

The final test: find the right time

Alright, this is all good and nice but we all know that often the hard part is not entering the data, but finding the right slot. Surely the model isn’t clever enough to find an open slot in my agenda… right ?

Ah, finally we reached the frontier. The model doesn’t read this right and wants to schedule the climbing session for Monday, despite having just helped me to schedule a cinema event for the same day.

Something is going wrong with reading the calendar, but it’s not just random hallucination: the model seems to read some events, but miss others:

In conclusion, we hit a classic LLM situation: some things work and it’s quite amazing that they do. But when they don’t, the model has no idea that it’s wrong and it’s hard to tell how to correct this.

I guess we get to live with the exciting reality of being responsible for our own time planning for a little while longer 💫.

In other news

• 🍄 Dall-E Party: ask Dall-E to generate a picture (e.g. a cat lecturing mice about food), then feed the image to GPT4 and ask it to Write a prompt for an AI to make this image. Then feed this prompt back into Dall-E and keep going in circles. It starts pretty close to the initial prompt and then becomes weird fast (see above)

• 🎞️ “Time is the thing” – memorable scene from Tár, source of the title headline

• 🏴‍☠️ GPT4 is trained to refuse solving captchas, but just like with pure text jailbreaks, people find ways to bypass the guardrails

• 📝 Recent paper from Microsoft on multimodal foundation models

Postcard from Paris

Wet rooftops of Paris ☔️. Perfect weather for staying inside, drinking tea and thinking warmly of people and things.

Have a great week 💫,
– Przemek