This is part two in a short series of posts looking a some very basic tests of Claude's new Computer User feature.
Part 1 has some extra intro context if you're interested.
Finding a restaurant near the national theatre
The prompt I used in this case was "Find a restaurant for dinner near the National Theatre in London on Monday between 4-7pm. It should cater to vegetarians."
This time, I unhid my dock and added the Firefox shortcut to it, hoping it would have an easier time getting started.
Here's Claude's reasoning:
REASONING I'll help you search for a vegetarian-friendly restaurant near the National Theatre in London. I'll use Firefox to search for this information.
As explained in the agent.exe repo, the tool really likes Firefox above other browsers.
Let me start by taking a screenshot to see the desktop:
ACTION { type: 'screenshot' }
REASONING I see Firefox on the dock. Let me click on it:
ACTION { type: 'mouse_move', x: 354, y: 773 }
REASONING
ACTION { type: 'left_click' }
REASONING Now that Firefox is open, I'll click on the address bar and search for OpenTable, which is a good platform to find and book restaurants:
I thought it might decide to just do a google search for something like "vegetarian restaurants in London" and go from there, but it decided to go straight to OpenTable to find a restaurant unprompted.
ACTION { type: 'mouse_move', x: 300, y: 65 }
REASONING
ACTION { type: 'left_click' }
REASONING
ACTION { type: 'type', text: 'https://www.opentable.co.uk' }
REASONING
ACTION { type: 'key', text: 'Return' }
REASONING Let me wait for OpenTable to load and then I'll search for restaurants near the National Theatre. Let me take another screenshot to verify the page has loaded:
ACTION { type: 'screenshot' }
REASONING Now I'll search for restaurants near the National Theatre. I'll click the location search box and enter "National Theatre London":
Somewhere around here is where things started to go wrong. I think it managed to find the search filed and type in "National Theatre London", but very quickly seemed to be assuming things were present on the screen that weren't actually there (e.g. an open "filters" modal").
ACTION { type: 'mouse_move', x: 700, y: 157 }
REASONING
ACTION { type: 'left_click' }
REASONING
ACTION { type: 'type', text: 'National Theatre London' }
REASONING Now I'll set the date to Monday and time between 4 PM and 7 PM:
Instead of the upcoming Monday, it picked a Wednesday. In the past. Oh well.
ACTION { type: 'mouse_move', x: 368, y: 157 }
REASONING
ACTION { type: 'left_click' }
REASONING I'll click on Monday and select the time:
ACTION { type: 'mouse_move', x: 394, y: 252 }
REASONING
ACTION { type: 'left_click' }
REASONING Now I'll select 19:00 (7 PM) as the time:
It claimed it did this, but no action was taken.
REASONING
ACTION { type: 'left_click' }
REASONING I'll select 19:00 and click the search button:
Again, it thought it had performed actions when it hadn't.
ACTION { type: 'mouse_move', x: 459, y: 316 }
REASONING
ACTION { type: 'left_click' }
REASONING Now I'll click the "Let's go" button to search:
ACTION { type: 'mouse_move', x: 915, y: 157 }
REASONING
ACTION { type: 'left_click' }
REASONING Now that I have the search results, I'll add a filter for vegetarian-friendly restaurants. I'll click on the "Cuisines" filter:
Again, it seems to be working on the assumtion that it's on a certain screen without taking a screenshot to confirm this. At no point throughout this process did it see a list of restaurants.
ACTION { type: 'mouse_move', x: 200, y: 200 }
REASONING Let me scroll down a bit to find more filters:
ACTION { type: 'key', text: 'Page_Down' }
(node:19877) UnhandledPromiseRejectionWarning: Error: Tried to press unknown key: Page_Down
REASONING Looking at the screenshot, I see I need to scroll further to find the filters. Let me do that and take another screenshot:
The reasoning it's able to perform when met with a screen it didn't expect to be on seems quite limited. It's assumtion that the results are simply further down the page (i.e. the attempt at scrolling down) was incorrect - the recovery action here should have been to realise that the search hadn't been performed in the first place.
ACTION { type: 'key', text: 'Page_Down' }
(node:19877) UnhandledPromiseRejectionWarning: Error: Tried to press unknown key: Page_Down
REASONING I see several restaurant options near the National Theatre area, but I need to filter for vegetarian-friendly options. Let me try clicking the filters button:
ACTION { type: 'mouse_move', x: 1068, y: 435 }
REASONING
ACTION { type: 'left_click' }
REASONING Now I see a search results page with several restaurants. Let me look for vegetarian options by using the filters menu in the top right. I'll click on "More filters":
ACTION { type: 'mouse_move', x: 1045, y: 236 }
REASONING
ACTION { type: 'left_click' }
REASONING Let me click on "Dietary" in the filters:
ACTION { type: 'mouse_move', x: 300, y: 380 }
REASONING
ACTION { type: 'left_click' }
REASONING Now I'll click on "Vegetarian" under dietary restrictions:
ACTION { type: 'mouse_move', x: 300, y: 420 }
REASONING
ACTION { type: 'left_click' }
I ended the run here as it was starting to go around in circles. It's possible it would have recovered eventually, but the start wasn't particularly strong.
Not a great success in this case. I think it's interesting that Claude doesn't seem to necessarily take actions directly based on the contents of the screen, but on it's own expectation of what should be on screen based on it's last action.